Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deepmind deep mind ml and society ai and society sociology and machine learning machine learning for sociology machine learning for economics ai microeconomics reinforcement learning economics society simulations silly rules social norms social norms enforcement why do social norms exist why do silly rules exist deep mind society
#deepmind #rl #society

This is an in-depth paper review, followed by an interview with the papers' authors!
Society is ruled by norms, and most of these norms are very useful, such as washing your hands before cooking. However, there also exist plenty of social norms which are essentially arbitrary, such as what hairstyles are acceptable, or what words are rude. These are called "silly rules". This paper uses multi-agent reinforcement learning to investigate why such silly rules exist. Their results indicate a plausible mechanism, by which the existence of silly rules drastically speeds up the agents' acquisition of the skill of enforcing rules, which generalizes well, and therefore a society that has silly rules will be better at enforcing rules in general, leading to faster adaptation in the face of genuinely useful norms.

OUTLINE:
0:00 - Intro
3:00 - Paper Overview
5:20 - Why are some social norms arbitrary?
11:50 - Reinforcement learning environment setup
20:00 - What happens if we introduce a "silly" rule?
25:00 - Experimental Results: how silly rules help society
30:10 - Isolated probing experiments
34:30 - Discussion of the results
37:30 - Start of Interview
39:30 - Where does the research idea come from?
44:00 - What is the purpose behind this research?
49:20 - Short recap of the mechanics of the environment
53:00 - How much does such a closed system tell us about the real world?
56:00 - What do the results tell us about silly rules?
1:01:00 - What are these agents really learning?
1:08:00 - How many silly rules are optimal?
1:11:30 - Why do you have separate weights for each agent?
1:13:45 - What features could be added next?
1:16:00 - How sensitive is the system to hyperparameters?
1:17:20 - How to avoid confirmation bias?
1:23:15 - How does this play into progress towards AGI?
1:29:30 - Can we make real-world recommendations based on this?
1:32:50 - Where do we go from here?

Paper: https://www.pnas.org/doi/10.1073/pnas.2106028118
Blog: https://deepmind.com/research/publications/2021/Spurious-normativity-enhances-learning-of-compliance-and-enforcement-behavior-in-artificial-agents

Abstract:
The fact that humans enforce and comply with norms is an important reason why humans enjoy higher levels of cooperation and welfare than other animals. Some norms are relatively easy to explain; they may prohibit obviously harmful or uncooperative actions. But many norms are not easy to explain. For example, most cultures prohibit eating certain kinds of foods and almost all societies have rules about what constitutes appropriate clothing, language, and gestures. Using a computational model focused on learning shows that apparently pointless rules can have an indirect effect on welfare. They can help agents learn how to enforce and comply with norms in general, improving the group’s ability to enforce norms that have a direct effect on welfare.

Authors: Raphael Köster, Dylan Hadfield-Menell, Richard Everett, Laura Weidinger, Gillian K. Hadfield, Joel Z. Leibo

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Why do social norms exist? And why are some of them really, really meaningful? And why do some of them make no sense at all? Like, why am I not allowed to wear this hat right here to a funeral? Okay, it might upset some people, but why? There is no benefit. There's no direct welfare impact to society with me wearing this hat. or not wearing this or wearing something else on my head. This is a question that we're going to investigate with today's paper. And yes, that has no inherent relationship with machine learning. But as you'll see, we can tackle this question or at least a part of the question we can give some evidence as to why these what's called silly rules might exist using machine learning, specifically deep reinforcement learning. So in this paper, people from different areas of expertise came together to say, why are some of these rules useful? And they said, why is Can we build a computational model of society? Can we build a little world of agents? Have them do some behavior, give them some rewards for certain things, and then we just observe what they do. And by observing, we can make some conclusions about, huh, this could be an explanation for a societal phenomenon that we see. So I like this paper because it's interdisciplinary. It uses deep reinforcement learning, specifically multi-agent reinforcement learning, in order to answer questions about society. And it is a little bit out of the box, which I like. So the video is structured. I first do a review of the paper by myself, and then I'm going to talk to the authors about the paper. This is one of the last videos where I recorded the interview before I did the review. But for this paper, it was actually super helpful because I'm a noob at this field. I don't know what I'm talking about when it comes to society and research in sociological questions. So it was very helpful to have the authors talk to me about the paper. But we don't just talk about the paper. We talk about many, many more things. And I highly invite you to watch the interview because it's really interesting. We talk about norms and societal systems of norms and hypotheses and what you have to pay attention to when you do research like this and what worked and what didn't and what it means. So please let me know if you like papers like this that are maybe a bit more distant from what we usually do. And if you do, then please let me know what other kinds of papers and what other areas exist where ML and specifically reinforcement learning or any kind of machine learning are used to investigate questions in other fields. All right, I'm going to leave it at that. And now I'll just do like a quick green screenshot because I know people are going to make emojis out of my face with this hat on. So. And that's that. Cheers. What they call silly rules. So the question is, our society has a bunch of norms of what you should do and shouldn't do. And these norms are known by the people and they are enforced by the people. You're being shamed if you don't follow the norms. A lot of those norms are really good, like wash your hands after you use the toilet. But there are a lot of norms that are also just arbitrary. Like what kind of hairstyle is good and bad or acceptable or not acceptable. What words are rude and things like this. And these are called silly rules. And the question is, why do these exist? Now, this is not a question of machine learning. However, this paper applies deep reinforcement learning in order to give some evidence to why these rules can exist. So I like the mixture here of sort of using reinforcement learning as a tool to investigate these mechanisms by using a computational model. You can break down a lot of things. Usually, if this were a psychology paper, people would go into a lab, they would recruit people, and then they would try to design an experiment around these norms and so on. And that's cool and all. But if you use a computational model, you can answer different questions. You can control for different variables and so on. So it's very attractive to use reinforcement learning for that. So we're going to look at what this paper says right here. Not as much into the RL part because that is fairly straightforward. But just what it does and what it says. And I'd like just to show you maybe a little bit because I thought it was pretty cool that this is yet another application of machine learning and specifically reinforcement learning that enables progress in a different field. So I hope you enjoy this. Yeah, they introduce the paper by saying there are a lot of norms. Something that differentiates human from other animal society is this presence of norms. And some of many of these norms, say, generate direct benefits for individual and group well-being, like, you know, reciprocity, sharing of rewards, what you should eat, what you shouldn't eat, and so on. Very often, these rules have some sort of a benefit to society. They say, but, however, the normative landscape is also populated by many norms that appear essentially arbitrary and without direct material consequences. And we're not necessarily fighting about this. Like, people can always say, well, but this rule may have some use. But let's just, for now, let's assume that there exist norms that really could be different, and it would make not a difference in total welfare, or at least a direct difference, right? The paper here argues that there is an indirect difference. The paper argues that by introducing these silly rules, the indirect benefits are that agents learn the enforcement behavior of the rules more clearly. And therefore are better at enforcing the important rules. But we'll get to that in just a second. So here are some of the examples of silly rules that they mention. Men are expected to wear pants, not skirts, which in some societies is the case, and others isn't, right? There are words or hand gestures that should not be used in polite company. There are rules about how one's style of hair or what one wears on one's head, and so on. So they call these silly rules. Silly rules means essentially a norm that is in society, is very, you know, taken seriously, but is essentially arbitrary. They say they're meaningful and enforced, but they have no direct first order impact on welfare. So why do they exist? There are some hypotheses. They list some here. They say, for example, silly rules may remain stable by virtue of their incorporation into larger normative systems that also include important rules, which essentially means that the silly rules, they make sense if they are part of a bigger system that also contains the important, which means the useful rules. And so the hypothesis here is that the addition of the silly rules into a society somehow helps the society to comply more broadly or more or more or better or more accurately with the important rules. So the addition might be some might be a benefit in the total benefit, like total setup of the system. In this paper, they say we describe a mechanism through which silly rules can benefit a society. Our argument is based on the dynamics of learning in a group that lacks a priori knowledge of which of the rules are truly important. So there is a group, there's a society, there are a bunch of norms already present, and a priori, no one can tell which ones of those are important and which ones aren't, because if they could tell, they could just say, well, that one is not important, which is what's happening kind of with the scientific method, right? We know that some things aren't as important and with time, people stop doing them. But initially, you know, there's no way of knowing. And that's what they investigate. It's important that they say, they describe a mechanism, right? They don't necessarily say this is how society works, right? Because society is way more complex, but they do describe one possibility, one mechanism, one reason why these silly rules could exist. And they show that this mechanism, if you implement this in a mini-society, will lead to a total welfare benefit. Their explanation is the following. The skills involved in third-party norm enforcement readily transfer from norm to norm, while the skills involved in compliance are norm to norm. The skills involved in compliance are norm-specific. What that means is, essentially for every norm, you have to learn how to follow that norm. So these are the skills involved in compliance. They are norm-specific. If, you know, there's a food I shouldn't eat, then I have to learn to avoid that food. And then if there is some sort of like a way, like, please share if you have enough, like that's a norm, I have to learn how to do that. For many norms, the skills to behave in accordance to the norm are very specific to the norm. However, the enforcement, this enforcement skills, they transfer from norm to norm. So what's the enforcement skill? For example, shaming someone if they don't follow a norm. That's very, that's similar from norm to norm, whether they don't follow the hygiene norms or the interaction norms or the food norms or the hairstyle norms is always the same to shame someone into compliance or to, I don't know, deduct from their social credit score or something like this. So they argue that the skill of enforcing norms transfer while the skills of following norms don't transfer as much. And therefore, they say, the silly rule may provide greater opportunity to practice third party norm enforcement. And through that, the third parties will also become better at enforcing the true, the useful norms. So the addition of silly rules might simply make it easier for people to learn to shame others into submission. And by that, they will be more effective at shaming them when it comes to the good norms, which obviously they don't know. So they're just going to shame for all the norms. But overall, it is positive in welfare. So what they do is they have this environment right here. You can see the environment right here. So up on up here is a schematic of the environment, but this is kind of the representation. They are going to have a map, which is a 2D map. You can see that right here. That's the map. And sorry, on this map, you have agents. So an agent right here, that's sort of a little person that's walking around. The person can walk around so they can walk up left, right, and so on. Every person sees a little window around themselves. They see what's happening around. There are sort of obstacles there, but there are also these berries. And the berries, I don't know if you can see them on the screen, but the berries, this is a berry. These are two berries right here. They come in different colors. So the agent's goal is to move around and collect these berries. Every berry they get, they get some sort of points. You know, they collect them. That's the reward. There are enough berries so that there is no meaningful competition between agents. There is one other thing they can do, and that's zap someone. They call it even zapping. So in this case, I'm going to guess something like this agent right here is zapping this agent down here. And the yellow thing is a punishing, punishing beam. Essentially, that just means that the agent can zap another agent, which will cause the zapping agent to lose a bunch of points and the zapped agent also to lose more points. The only addition now comes with the poison berries. So sometimes some of the berries are poisoned and there will be a color selected for which berry is poisoned. For example, let's call all the green berries here. They're poisoned when an agent picks up a poison berry. They are they they won't see necessary. They won't see it themselves, but they will be poisoned. And after they pick up a poison berry, 100 steps later, they will start to lose health or I think they will just they will not gain as much from eating other berries. That's it. So there is a very delayed, very slow punishment for eating poisoned berries that takes the agent a long time to learn that. However, if now if you get zapped while you're poisoned, that gives the zapper a benefit. So let's call this person Alice here and this person Bob. If Alice zaps Bob and Bob is fine, then Alice loses some points and Bob loses some points. However, if Bob is poisoned, then Alice gains a bunch of points for zapping Bob. So Bob is poisoned, loses points and Alice gains points by zapping Bob. I do think so. The zapping cures Bob, I think. So one zap will actually cure Bob, but Bob loses a lot of a lot of points. Hey, y'all, it's Yannick from the future. I made a small mistake right here in that I claim that zapping cures the poison, which it does not. The idea is that zapping removes the mark. So when a player eats a poisoned berry in this normal rule condition, they become marked and zapping cures the mark. If you zap a marked player, you get points, but zapping removes the mark. It does not cure the poison. The poison is still active. The idea is obviously that the players learn to avoid the poison in the first place because they don't want to get marked because they don't want to get zapped. And now in the silly rule condition, also a second berry activates the mark, but that's not a poisoned berry. And this you would expect that it's more noisy and therefore learning is more difficult. But it turns out under the silly rule condition, learning is actually more efficient. And that's kind of the point of the paper. So again, the zapping doesn't cure the poison. It just removes the mark in whatever way that mark happens to be on the map. Happens to be on the player in the first place. Back to the video. Yeah, there's one last thing and that you can see here in the marking. So when an agent is poisoned, so when they after they've eaten a poisoned berry, they become marked, which means that all the other players will see that they are poisoned. Now, this is the setup. What you can pretty quickly see. So no rules is here. We have berries and we have poisoned berries that give you a delayed punishment. Then this is what I just described with what's called the important rule condition, which is that if you eat a poisoned berry, you become marked. And then if a third party and other players sees that they can zap you and they gain a bunch of points. So you can see that pretty quickly. What is going to happen is that the agents, they learn to eat berries, but then pretty quickly they learn to spot the marked agents and they zap them. And then after that also very quickly, the other agents will learn to avoid the green berries because they realize wait, every time I get a green berry, I get zapped later. And that's how that's how the agents avoid learn to avoid the green berry. Note, we have to clarify some things. This paper isn't about how the norm of not eating the green berries comes to be because obviously that's kind of like God given right here. The marking is done by the environment. The rewards are clearly set up such that people learn to avoid the green berries. That's not the issue right here. The question that the paper has is how quickly can the agents learn to enforce that norm? So how quickly do they catch on zapping others? Right? And what is the overall welfare? So the norm itself is set by the environment or by the designers of the experiment. We are not trying to learn to avoid the green berries. We are trying to learn to avoid the green berries through the effect of poison. But we simply directly give rewards for zapping the marked agents. And that means we... Deus ex machina... Ex nihilo... What means just like we command a norm onto the system and we see how the agents react. So that is obviously what's happening here is not a secret. Imagine that by the way the agents they use an actor critic. They use a simple conv net and an actor critic framework to learn right here. What I find interesting is that there are 12 neural networks. So the system keeps 12 neural networks that are initialized with the same weights, but they're different neural networks. And 8 of the 12, I'm gonna just select three or four right here, but imagine that's 8 of 12. 8 of the 12 are then each episode drawn to compete in the ring. They compete for a thousand time steps, then they get their learning updates, they get put back and then for the next thing 8 others are drawn. Which I found pretty interesting. It's a way to sort of get diversity into the system. Now what does that have to do with silly rules? So far we've built up an environment. We forced a norm onto it by giving reward for punishing these marked agents. And we've discovered that agents learn pretty quickly to enforce that norm, which in turn makes all the agents avoid the poison berries as a consequence of being punished by the norm. Now we introduce this silly rule. So the silly rule means that there are poisoned berries, which are these ones, but there are also other berries that we will call taboo berries. The taboo berries, they're just fine. They're just, you know, they're fine. They're healthy. You can eat them. You get a bunch of points for eating them. That's fine. However, if you eat the taboo berries, you will also become marked, just like the poison berry eater. Right? So these are indistinguishable markings. And therefore, the agents that learn to gain points by zapping the taboo berries will also gain points by zapping the ones that ate the taboo berries. What's even worse is that they also get reward for zapping the taboo berry eaters. So there's no difference in the reward for zapping that you get if you zap a poison berry eater or a taboo berry eater. You just, whenever you zap a marked player, you get some points. Again, it's not about how the agents learn to avoid the poison berries. It's how they react to given norms. Right? So again, we enforce the norm of you should eat neither the poison berry nor the taboo berry. Of course, the agents don't know which one is the poisonous one. They just know they get zapped after eating either the pink or the green berry. So how does that go? That's sort of the question of this paper. We've introduced a silly rule, which on a surface serves no purpose. The green, making the green berry taboo serves no purpose other than it's just, it's just a rule and you get punished for not following it. It even decreases the overall welfare a little bit because now you don't want to eat the green berries anymore, which means that you don't get as many points. The question is, can the introduction of the silly rule get you an overall reward? An overall benefit as a society? That's the question. So we'll go on a little bit. They say our model allows us to separate the learning of enforcement and compliance behaviors from the learning of the norm content itself. That's what I repeatedly emphasized because I had a lot of trouble when reading this paper to really get this. They don't want to, they don't want to, they say here, we designed an experiment in which norm content was fixed in advance by the experimenter, namely which berries are taboo. The question is, how do they react to it? So this is a brief recap. If a player breaks the taboo, they change color in the observation of other agents viewing their transgression. They become marked. If a player is marked, other players can collect a reward by punishing them. This creates an incentive for players to learn to punish rule violations and thus for players to learn not to violate the rules. And these are the results. We show that individuals achieve higher overall welfare in a world where eating the poison berry is taboo. That's condition one. This is clear. This is logical. We take a delayed punishment for eating poison and we essentially bring it to the present by having people zap the poison people and them learning to avoid it. However, the main results, sorry, they say even with the cost of enforcement, overall group welfare is higher with the norm than without. We then show our main result that the value of the normative order is higher if the set of norms in this regime includes not only important rules such as the rule against eating poisonous berries, but also silly rules which make the eating of a harmless berry taboo and bring about the same third party. Punishment. So they show there is a situation right in which you can gain by introducing such silly rules because enforcement skills are learned faster. Let's just quickly look at the agent architecture. If you're into machine learning or RL or so, this should be rather familiar to you. So the agent, they see raw pixels up here. There's a neural network. It's a CNN followed by an MLP. There is an actor critic. So there is a value function and there is a policy function. Actor critic, very basic actor critic algorithm. This is obviously a very easy environment for reinforcement learning and that makes it ideal to use multi agent RL here to gain some insights. As I said, we have 12 agents, 8 out of 12 play in 64 environments in parallel. And they get the replay buffers and they update those weights. All right. Yeah, I've mentioned these things. I've mentioned these things. Now let's look at the results. So first of all, let's look at fraction of time spent poisoned. Like how? So here is time step strain. So this is over the course of training. Right. So what fraction of the time do the agents spend? Does an average agent spend poisoned? If there is no rule, you can see that there is a constant fraction of the time agents spend poisoned. Essentially over the course of this training, they don't learn really to avoid the poison berries and therefore, yeah, because the reward is just too delayed. I guess the RL algorithm also isn't too powerful, but you can see that there is a clear difference between the important rule and the silly rule. So important rule means there is only one rule, shouldn't eat the poison berries and silly rules that means that there is in addition this silly rule. So the agents here quickly, they spend less total time poisoned. And the question is, is why? So let's look at some other effects that the introduction of the silly rules have. Total taboo berries eaten. You can see that at the beginning, about double the amount of taboo berries are eaten under the silly rule than under the just important rule, which makes sense because twice as many berries are taboo. So you'd eat twice as many of them in the same time. But you can see that there is a crossover. This decreases and there's actually a crossover. So after a while, less taboo berries are eaten than in the important rule setting, even though there are more taboo berries, right? So somehow these agents learn faster to avoid the taboo berries. Total punishments. Now, obviously, again, at the beginning, there are double as many taboo berries, so double as many marked players. So they go, the number of punishments goes up pretty quickly. And then there's a crossover point where after a while, there is less punishment going on than in the important rule. So these societies, they learn faster. And that's, I think, the point. You can see that at the end, there's often sort of the same result, the same outcome, but in this intermediate stage. And remember, society is always in flux, kind of. So one can argue that very often we are at all times in sort of this intermediate stage. So in this intermediate stage, it's actually an overall benefit. Fraction of time spent marked goes down as well pretty quickly, obviously, because people are more marked. And collective return. So here is the actual result. If you have no rule at all, collective return goes up at the beginning, it's actually the highest, but then flat lines, right? Because people keep getting poisoned and that hurts. If you, however, use this important rule thing, then at the beginning, it's not as great, because if you punish, the rewards are structured such that if you punish, you decrease the total welfare. Even though you as an agent gain some points, the total number of points in society decreases as a result of punishment. So you can't just punish more and more and more and expect to get more and more. You have to expect the collective return to grow. So yet still, because agents learn to avoid the poison berries through punishment. So at the beginning, there's lots of punishment. That's why the reward, the collective return is lower, but then they learn. And as they learn, they learn to avoid the poison berries, then they don't need to punish as much anymore, right? And then the reward goes higher than if you had no rule at all. Most interestingly, however, in the case of the addition of the silly rule, you can see that at the beginning, there is a decrease in collective return as people punish around, like they punish each other to death. Yet, yet, very quickly, this goes up and actually becomes the highest collective return there is. And you can see in this intermediate period right here, there is clear benefit to having these silly rules around because the society is much quicker and much better at learning to avoid the poison berries because, because, and you can see from the time series right here, because they learn much more quickly to punish, to punish people who eat the wrong berries, not only the poison, but also the silly ones. And because they're much quicker at punishing, the agents have more opportunity to learn to avoid these berries, and that's what gives you the higher return. They do investigate what these agents have learned. They say psychology experiments with human participants address the issue of learning what people have learned individually by isolating specific mechanism and testing in these controlled conditions, such as reactions to particular stimuli. They want to do the same thing computationally. So they take these agents from their training run, they put them in inference mode, and they give them like a little environment like this. So they start apart from the berry and the episode ends on contact with the berry. So then there you can give them a berry and see if they eat it or if they don't eat it. So if you have no rule at all, if you don't have this marking rule or anything like this, here again, it's time steps trained, but remember, we don't train the agent on this task, we train it on the original task, then at certain checkpoints, we take it out, we put it in little lab and we see what happens. Also, the y axis here is inverted. So 30 is down here, which means 30 time steps. If the line is here, it means the agent has not eaten the berry. If the line is up here, or like somewhere up here, it means the agent has immediately eaten the berry. You can see that if you have no rule, agents, they just eat the berry. Doesn't matter if it's poisonous or not, right? The pink is poisonous. It makes a little bit of a difference, but not really. They just eat it. If you add the important rule, they quickly learn to avoid the poison berry. You can see that right here. If you add the silly rule, they also learn to avoid not only the poison berries, but also the taboo berries. They also, in fact, learn to avoid the healthy berries a little bit more, but this comes back over time. There is a bit of an unlearning right here, and I do ask that in the interview. They specifically highlight... So these are different berries. Now, just isolating the times when they give the agent a poisoned berry, you can see that the reaction to the poisoned berry is much, much bigger if you are in the condition that contains the silly rule compared to if you're in the condition that doesn't contain the silly rule in this intermediate regime right here. And also, you know, the punishing is way quicker. So they measure how long it takes you to punish. It's way quicker when you have the silly rule. So that's essentially the evidence that they say, look, these agents, they learn the skill of punishing. They learn the skill of running after someone who is marked and therefore punishing them. And that gives the agents the opportunity to learn to avoid poisoned or marked berries altogether. And because there is more punishment, because the agents are better at punishing more early on, they learn to more quickly avoid the poisoned berries. So the overall argument again is that the skills of punishing are transferable between tasks and the addition of a silly rule, even though it brings some negative welfare because it's a rule you need to follow, like you incur some cost, it could still be total benefit overall because the introduction of the rule just trains people in punishing others for not following the rules and therefore trains people in following rules and therefore trains people in following the important rules. Remember, in this society, people have don't know, the assumption is they don't know which of the rules are beneficial and which ones aren't. So these were in the discussion now, they say from the perspective of an agent learning the skills necessary to effectively enforce their society's norms, the additional violations constitute additional opportunity for practice, and thus promote a faster rate of improvement in their command of the mechanisms, sorry, of the mechanics of third party punishment. Now obviously, this doesn't go forever, right? You can't just add silly rules until you know, like until the world is just made of rules and expect well, we're always going to have much higher welfare. But there is a regime where that is the case, and we might as well live in that regime in our societies. They say enforcement and compliance are asymmetric in the sense that the former is a skill that may be applied without modification to any norm that's enforcement. Since many of the sub behaviors involved in third party punishment are directed towards the violator, for example, chasing them, not towards the event of the violation itself. Thus, they are transferable skills generically applicable to any norm. And yes, I get it if you say, for example, avoiding food is also transferable and so on. Sure, sure. But I think this sentence here that a lot of punishment behaviors are directed towards the violator and not towards the event of the violation itself, that it makes sense that these skills are more transferable. The interpretation of our key result is that the role of silly rules in human normative systems may in part be to help train a society's ability to comply with important rules. And that is the result. The paper goes into more detail, obviously, in all of these results in the setup in why it's important and so on. But I'll leave it at that for now. I hope you gain some insights into how reinforcement learning can help other fields to get some insights by modeling sort of these computational little societies and just introducing aspects of the real world. And then just seeing how that pans out. It wasn't clear at all from the beginning that the introduction of the silly rule here would bring this improvement in sort of the intermediate timeframes. And that's just really interesting. And it's kind of a different way of approaching the questions of why does silly rules exist in society. Questions like these, it's a different way of approaching them than just putting some humans in a lab, which has its own problems, right? So I think this just gathers some evidence and it's pretty cool. And it's an opportunity for interdisciplinary research, which I like. And I hope this was fun to you as well. And I'll see you around. Bye bye. Hello everyone. Today I have with me here three of the authors of the paper about spurious normativity enhances learning of compliance and enforcement behavior in artificial agents, Gillian Hadfield, Joel Liebow and Rafael Custer. You are an assembly of people with way different backgrounds that have somehow come together and focused on a very cool intersection between machine learning and social sciences. Welcome to the channel and yeah, welcome. Thanks for having us. Great to be here. So I mean, the first things first, in machine learning, we've had these trends of just making like clickbaity titles. I feel your field should pick that up because a title like this, it's like that is an instant desk reject. You got to have like a little acronym, like spell or something, like just four letters or so and then, or a question. But yeah, it's a pretty cool. Yeah, it is. We did have a somewhat more intriguing title than the journal told us to change. Yeah, we did have silly rules in the title for this reason and they were nervous about that. Okay. There is still some veneer of professionalism in other fields of science, not in ours. Yeah, I was very, very happy to see this paper because it connects something that I know to something that I don't know. And I think, you know, us machine learners were sort of always in the same areas. And this goes a little bit outside of my comfort zone. So I thought it was pretty cool. How did you get like the idea of writing something like this, of connecting these fields? Like where does it come from? I can start with how I came to it. So my background is in computational neuroscience. That's where I did my PhD in. And when I came to DeepMind, I was thinking about how do we build artificial general intelligence and reading lots of things about human intelligence and realized that intelligence isn't really in the brain. So my whole PhD on neuroscience was maybe not as helpful as I thought it would be. But intelligence is actually a collective phenomenon that is more supported by how societies work and how we cooperate with each other and learn from each other and things like that. And so since then, I've been trying to build human like AGI in a way that is more like trying to make a society of AGI. And this was one piece of work that came out of that after meeting Jillian. Maybe Jillian can speak. Yeah, maybe I can say a little bit. So I'm a social scientist. I don't build these systems. I think about and study how human normative systems work. Right. Those are our systems of norms and our systems of rules. And I'm very interested in that from a systemic point of view. What are the attributes of the systems that make them stable and adaptive and contribute to human progress and evolution? And so I've been thinking about working on those kind of models, these economic modeling tools. And Joel's team at DeepMind had produced some papers studying some very standard problems in the economics literature on like tragedy of the commons and showing how they could use sort of those multi-agent reinforcement learning setups to study tragedy of the commons, which is sort of econ 101. I saw those papers, got very excited and said, oh, but we could really dramatically increase the sort of the social science component of this work. And I had been working with Dylan Hadfield-Minell, who's also on this paper on this concept of silly rules. And so actually, I think I tracked you down, Joel, and started a conversation a number of years ago. And we gave a talk. Yeah. We spoke afterwards. Yes, right. Oh, that's right. I came and gave a talk at DeepMind. And yeah, so I was very excited to be connecting up these two worlds. And then you needed someone to actually do the work. And then that's where Rafaela came in. I think I don't have much to add to Joel's story. So my background is also in cognitive neuroscience and psychology. And I work on topics that are sort of on the intersection of decision making and memory in humans and in AI. So social cognition, as well as learning from others or how groups behave is similar. And also questions of behavioral economics are all sort of all in the scope of what I'm really interested in. So I think this is, yeah, like a good example of where these things come together. Yeah, it's pretty cool. So to give the brief introduction to maybe the paper, I think it's maybe for the machine learners it's valuable to start with this one right here. So we have this environment. There are different agents inside of it. I think you already always have eight agents that take part in an episode. The episode can go up to like a thousand steps. In each step, each agent has the ability to move around. The goal is to collect the berries. It has like a little window view around itself of the world. And there is one other action. It can like zap someone else, right? It can zap, punish an agent. And we'll get to that in a bit. So these berries that are around, you deliberately made the berries plentiful. So there's no issue of like, yeah, competition or anything like this. There are three conditions that you compare and these are kind of your experimental conditions. Do you want to maybe say like, if you gave the pitch about your own method, I think this kind of is the core right here. How would you describe it? I might want to say what the purpose was. Yeah, sure. Experimental conditions, right? From my perspective, one thing that I think following on from what Jillian said a minute ago, it's true. We really did have a bunch of papers that were kind of reproducing economics 101 kind of ideas about a tragedy of the commons and things like that. And we had a sequence of those papers. And this was the first time we were really trying to like contribute back and say something actually new. That's not just like a new way of coming to the same kind of results that people already had in economics for centuries. And so this particular area we're trying to connect with is a field that's interested in cultural evolution and cumulative culture and things like human uniqueness. They see humans as an ultra social species. It's like critical to the niche that we are in. It requires a it's a cultural niche. We learn from each other. That's how our technologies work, how our societies are put together. And that's what's what makes us different from other primates. And so within that literature, one thing that's interesting is how is how we cooperate. And social norms are one kind of mechanism of cooperation. There's others like reciprocity and things like that. And then within that field, there's another question of like, we have all kinds of social norms, some of which seem to be relevant to cooperation, and some of which just seem to be irrelevant things. Like we can have a we can moralize all kinds of behaviors like you're supposed to wear clothes and you're not supposed to wear a hat in this circumstance or whatever. And the question that is like, well, social norms are so important for cooperation. Why are there all these other social norms that are like, just not doing that? I mean, is you have this concept of the you have this concept of the of the silly rule, right, which is a fantastic name. And it describes sort of a norm that isn't directly valuable to anything that that considers like group fitness or even personal fitness. Yet, does this actually exist? Like is there a rule where we can conclusively say this is a silly rule and not, you know, we might be missing some hidden advantage? Well, that's the point. You can never say that for any rule, really. Because you're inside the system, you never know whether this is there for some important reason or not. But I think this is a key thing is sort of just to sort of place this work in the context of the work that gets done on trying to explain human rules and norms. And so we have people come at this mostly from a functional point of view, like it's a solution to a game theory. It's a solution to a coordination challenge, or it's a solution to like a hot dove type problem where we're going to waste resources fighting over something that or cooperation, like Joel was saying, right? So most of our work in social science has come at the question of explaining norms by saying they serve this functional purpose. But it seems very clear we have lots and lots of rules where you could say, look, nothing would be different from a functional point of view. If we said you wear bright stripes at a funeral instead of black, or that you stand this far apart rather than this far apart. It's just once you start noticing silly rules defined in this way as no direct impact on welfare. Only impact, which is what we're showing, is the role those silly rules play in helping to stabilize a system by which people can enforce the important rules. So I think that's a key thing. So it sort of starts as a puzzle. Here's this thing that seems to be true of every human society you look at. Food rules, right? What we eat and don't eat is often a good example. Very tons across different groups and communities over time. Why do we have them? Why are they stable? There's really no good explanations in literature. So we got really interested in thinking about the role they play in supporting what I'd call the normative infrastructure, which is what you draw into enforcing important rules. If you're going to punish people for stealing your stuff or punish people for going back on their contracts, you need to have coordinated and incentivized your community to enforce rules. And what we're looking at is what's the role of silly rules in helping to create that structure. It is a bit like the value of just having rules. And if you have more rules, then you'll be better at following rules and people will be better at enforcing rules. And it's just like more rules sort of lead to... Because rules are a transferable skill. It's the enforcement part. And that's what you would want to get at right here. So your goal is sort of if we train agents and if we introduce like a silly rule like this, this skill would sort of transfer to beneficial rules whenever we actually have beneficial rules. So in the first context here, there are berries and there are poisonous berries. If you eat the poisonous berries, some when later, you'll kind of die, but your reward will shrink from eating new berries. So it will be like a very delayed thing. And in this case, we all know reinforcement learning isn't really good at super long rewards. You also have a discount factor, right? So the long rewards don't even matter. I could even imagine if a berry is close to me and I knew it was poisoned, I'd be like, meh, right? It's a hundred steps away. Who cares, right? I'll just eat it and I'll go back. But let's assume the agents actually want to avoid that. And then you have a silly rule and an important rule. The silly rule being you can mark or the rules are you can mark agents, right? Agents are marked. If you eat a berry that is taboo, you get marked. So you change the color and the perception of the others. So you yourself don't see it, but you change color in the view of the other agents. And if you are marked, other agents can collect the reward if they punish you. And so what we're doing with these three different conditions is we're sort of fixing what the norms are. That's the sort of the experiment is if you set the norms, what are the effects downstream on the ability of the agents to learn to enforce those norms and to then comply with the underlying rules that they are representing. And in the important rule condition, the taboo berry actually coincides with the one that is poisonous. So that's a really important rule for your group to have that should, if everybody learns to follow it, lead to everybody avoiding getting poisoned. In the silly rule condition, you still have the important rule. But on top of that, you also get marked for eating a berry that is fine and doesn't actually poison you. So there's the potential for twice the amount of transgressions and then also punishment behavior following that. The important thing is you get marked just the same. So in the third condition, whether you eat a poison berry or the berry that's fine, but just marked as taboo, you get marked the same. So there's no distinction. And the others collect a reward, whether you're poisoned or not, it's enough that you are marked right. So that that is how you sort of set these norms in place. Because I was I was sort of like, okay, the agents I have to figure out which one's poisoned, like no, they do get a reward as soon as soon as they zap someone who is marked. And now we're going to see what happens in a little bit as a result of these experimental conditions. But my question first is a motivation to punish those who have transgressed normative code and you want to like those those ones, they violated it, we want to enforce on them our social ethic or whatever. The question is a little bit. So there is this is like a microcosm, right? Sorry, there's a cat right here. This is a microcosm system. And I you know, there's always this in economics, there's always that the micro economists versus the macro economists, right? They and they and they kind of fight because the micro economists, they come up with their models and their simulations and their formulas. And then the macro economists are like, well, if you actually look at the whole world, it's completely different, right? Maybe you can get some insights, right? But there's always this danger of, you know, this enclosed system with these very constrained things. As soon as you introduce something else, it might just change the entire game. Is this something that you're, you're kind of avoiding somehow or worried about or not worried about? Should I take that one as the economist in the in the crowd? So I think there's there's a way in which what we're doing is the same kind of thing that micro economists which I am are doing, which is looking at, you know, idealized or schematic settings and doing theory about that in order to gain insight and generate testable predictions. And you're not trying to say this is a map of the world exactly as it is it's saying we can gain insight into what would be the impact of changing that price or that cost or increasing competition, that kind of thing. And so I think what we're what we're doing here is and we refer to this as kind of micro foundations, which actually lots of macro economists are interested in micro foundations, which is, is can we do a simulation like this to solve a problem that we can't do closed form with our theoretical tools like we would normally do like, you know, solve for an equilibrium or solve for, you know, a solution to a game theoretic problem. This is allowing us to solve a much more complex problem and gain insight and then demonstrate this, you know, we've got this hypothesis that said our agents will learn faster and better to both enforce and then therefore comply with rules if there's a silly rule in the environment. So I think a bit is kind of similar methodologically to that. I think it's got this this relationship to cultural evolution, not exactly one to one. We don't think humans started off like only being able to recognize pixels in the world, but that the idea that this is something that evolves over time, but we're not trying to kind of model like evolutionary game theory tries to in some ways model what would happen with repeat populations over time. So that's how I think about it. Well, I think it pays that we now jump to the results a little bit to take it ahead before we discuss sort of the like broader implications or anything like this. So is it fair? Like correct me if I'm wrong. I would characterize your main result or your main thing you derive from it that if I impose the taboo on the poison berry through this mechanism of agents getting reward, zapping each other, the population will sort of learn to avoid the poison berries better if then if if they just get the delayed anti reward. In addition, if I now also introduce another taboo berry, that's fine. It's silly rule, right? The agents can collect even more reward by by zapping, you would say they are learning the skill of enforcing rules, which is a generalizable skill. And through by becoming better at enforcing rules, they're sort of faster catching on to the fact that, you know, I should punish people for eating the wrong things. Therefore, the whole population learns to not eat these types of berries faster. Is that about in the ballpark? Yeah, there's there's an evolution of like the skills or what has been learned. Like at first, the agents need to learn to even perceive the world and then effectively eat berries that then increases to them actually getting poisoned a lot because they eat the wrong very a lot. And once that is in place, and you actually have a lot of marked agents, then it is possible to learn about the punishment and that it's that you can collect a reward for punishing marked agents. Once that is in place, then you have the opportunity to actually learn to avoid the berry you want to avoid because you are avoiding the punishment. But for that, you need all of the other agents to have learned to actually discourage this behavior. So this is sort of the nice progression of that one skill relies on another skill having been learned beforehand. And the silly rule helps exactly in providing more observations and more training for that learning of skills. And this is the sort of result you could only get with a model that is really focused on learning of skills. Another thing, another aspect of it is there's a very long temporal credit assignment problem, which is very difficult for reinforcement learning in the case where there's just poison berry. But in the case where they're being punished for eating that berry, then you're moving closer in time the negative thing to the event. So it's much easier to learn about it. This evolution you mentioned is visible in the graphs, right? So you first have like the total the total taboo berries eaten, it kind of goes up at the beginning because you get a reward for eating berries, then people learn to punish others, right? So that in time, you see that spike after the other spike. And then the like various things happen like the fraction of time spent poisoned and the fraction of time spent marked, they go down dramatically as a consequence of the punishments increasing. And at the end, sort of the collective return goes beyond what you would just have. So the difference here, I guess, is the credit assignment problem difference. There doesn't seem to be too much of a difference in the end result. Like if you let the game play out between the just the good rule, let's say and the silly rule. What is like so your claims are more about the evolution of the thing and somewhere in the middle, there might be an advantage to having the silly rule. Is that? Yeah, I was gonna say I think that's that's what's emphasizing that it's about learning these behaviors of, you know, the relationship between what you eat and Oh my god, somebody showed up and that's me. Right, learning that and then learning Oh, I get this reward if I zap somebody who is marked. So learning those behaviors, you know, once they're once they're learned in a stable, stable way, then the benefit of the silly rule is kind of okay, we've accomplished our learning objective. My own intuition is that that that the silly rules are going to help you with robustness so that when the environment changes, right, and they got to learn something new so that even though in our environment, it they they converges at the end, my guess is you can then introduce kind of the shock of you know, the rain didn't come this year or a different we're in a new part of the world and there's a different dangerous berry. Then then so I think that's that that that's likely if you sort of did follow on these experimental results, you have some more you draw this conclusion that what is the common thing is sort of the mechanism of enforcing rules. The agents they they learn this, this is a transferable skill. And by having sort of more taboos around, they learn this faster. What is different? Like what differentiates this hypothesis from the hypothesis that agents are better at avoiding some color of berry because by introducing, you know, a new taboo berry, I teach the agents that you know, this new berry is also taboo. And I say with the same argumentation that it may be not the enforcement that they learn in common, it may be avoiding some color of berry. Well, that's sort of the consequence, right? That's the compliance part. Yeah. From there, they can't see anything different until someone has enforced something on them. Because if they need a berry that is taboo, they're marked only in the eyes of others, they can't see themselves. And for the silly rule, nothing happens at all. It's just that they ate the berry and it became marked in everyone else's eyes. But from that perspective, nothing happened at all. So there's there's no effect on them in any way until the punishment comes first. Okay. Yeah, that's the only way that they could ever learn to comply. Is there a... And that's one of the nice the graphs in there to Rafael, the sort of showing that it is that sequence of learning to punish and then learning to avoid getting getting poisoned. A social equivalent to getting a reward for punishing someone who has transgressed a taboo. If I think to myself, the progression of this would be it would be more like if I enforce some taboo, then long term that will lead to more group welfare because everyone keeps to the rule, we eat less poisoned berries or we follow rules in general. And there is an aspect of group fitness that also reflects on me. You chose to directly give me reward if I punish someone for transgressing. Is this purely just because you wanted to like hard code these norms? Or is there like a social equivalent to that? Yeah, I'll take that from one perspective. And then I think we can do it from a few different ones here because this has multiple kind of ways of thinking about it. So the one you can see it as an intrinsic motivation agents just are motivated intrinsically to punish the transgressions of their norm that they have. So it's like some kind of like righteous anger on the part of the agent that just saw this this transgression. And then they're motivated to punish it. And that's a very kind of natural human emotion that we all feel for different norms. Like we could have totally totally different norms in mind, we can from different cultures to different places, but we might still feel a feel some like this is a transgression that we've just witnessed. I think it's whatever it is. That's one interpretation we could have. We have several others. There's this interesting one about medieval Iceland, maybe someone could say. Yeah, let me let me jump in there. So so so the fact that humans have this capacity for that they have this practice of third party punishment. So that's that really is distinctive about humans in the evolution of species. And it's a great puzzle. Why do humans spend resources punishing people for, you know, doing, you know, committing harm to others? It's that third party piece. And so we've got people in, say, behavioral economics who think it's about altruistic punishment. That's a little bit of what what the way I understand what Joel was talking about with intrinsic motivation that you just have a taste for punishings. We got a whole bunch of in behavioral economists who study sort of like, you know, people willing to pay money to be able to punish people for hurting other people. But it's a real it's a real puzzle in the story of cultural evolution about where that comes from. And so we have people who are in second order, like we have we have punishment for people who fail to punish. So we do actually have critiques that say, hey, how come you didn't say anything when that person said that harassing thing to the other person around the meeting table? Right. We have reactions to people who don't respond and don't punish people for violating our contract rules. And and in this anyway, it's a real, real puzzle. And we're hard coding it here. Some evolutionary anthropologists model it as a trait of punishment, like we have punishers and non punishers. My own view is that that's actually that that's the fundamental behavior to try and explain why do we end up with humans willing to spend personal resources punishing on somebody else's behalf, because that's the secret of our success. I was species. And should we do the medieval Iceland example? That's what that one's. Oh, oh, many of the lights. Yes. Right. So don't refer to the fact that I sort of been around looking at it really is about decentralized punishment. So the key thing to know about medieval Iceland is they had lots and lots of rules and they had no enforcers, no public enforcers, no police, no soldiers, no chiefs who had any power. They just have one individual, the law speaker who was responsible for reciting all the rules every year at a big gathering and who was the person you can go and ask, is this allowed? Not allowed. And that coordinates everybody on being willing. And they had very clear, not only rules, but what you could do, but also the penalties. Like if you did this, you had to give up 10 sheets. If you did that, you got kicked off the island. And what you need to do is coordinate your community to actually implement that punishment. And that's what they did really very effectively with zero public enforcement apparatus. Now eventually it becomes more efficient to have some enforcement apparatus, but individuals enforcing the rules is a really big part of both human history and even today really important. Think about mask mandates. Think about our pandemic rules. We're relying very heavily on community enforcement and non-enforcement. So the conclusion, the general conclusion is introducing a silly rule sort of makes group welfare higher or achieves the welfare faster, let's say by mechanism of, I learn a transferable skill and so on. So adding one silly rule, good. Adding two silly rules, adding three, adding four, like at some point, there must be a detriment to having only silly rules. How far would this go out? Is one the optimum? Is there some optimum of silly rules? Is this known? Can you assess that maybe with your simulation? So we haven't specifically tested this, but I think your intuition is right that there would be an optimal number because also every rule introduces costly effects because overall someone punishing someone else, overall destroys reward. So you end up with a net negative. So the more punishment there is, it's overall worse for the group. So the benefit needs to be quite large to overcome all of this additional punishment. So I think it would depend on how hard is, so first of all, how costly are they? If they're very cheap, then you can get away with more. The other thing is how hard is the thing that you're trying to learn? If it's very difficult to learn the punishment behavior and you need lots and lots of additional observations to do so, then I think additional rules would help. Whereas if it's very easy to learn, then you barely need any additional observations and you're just stuck with the bill. So I think it depends on that. I think it's some sort of inverted U shape with some optimal amount. I see in these graphs a little bit that sometimes at the end, actually trends reverse a little bit, especially in the silly rule case. And I've seen it here and here. It's also prominent in these sort of single agent tests which you do, which I really like. You take a single agent, you put it in a controlled environment. It's not training, it's just at some point during training, it's like an eval set. But also here, you kind of see these sort of reverse trends as training progresses. What happens there? Are they becoming really good? Do they learn the actual reward of being poisoned? Or what's going on there? Do they learn to avoid the punishers? I suspect that what happened there is some amount of unlearning because if you are very effective at teaching the population to not get marked and they effectively avoid all the taboos, then this behavior just doesn't occur anymore. You will just forget that you've ever learned that. So I think if this were to keep running, they might have to at some point relearn it. But then the question is if they actually would relearn it because now they have competition from different things. Maybe they're very good at collecting berries now, so maybe they're not as interested anymore as even learning about the punishment dynamics at all because the counterweight of their other behaviors is different. So I think this turns into a continual learning problem if you just let it run for a very long time. There's a covariate shift when the behavior of marked agents existing and then being available to punish is very different. Your structure has a bit of a special thing in it which I found, which is that you have 12 different agents, let's say 12 different neural networks that you train. In every episode, you choose eight of them to compete, whereas sometimes or a lot of times in multi-agent reinforcement learning, I have like one neural network, maybe with a bit of randomness, but essentially every of the multi-agents has the same weights. Let's say they're all shared. Was there a particular reason why you chose this specifically? Not only having different neural networks for each agent, but also to always sort of select subsets of them. And also, the follow-up is have you discovered that they diverge? I would be interested, did one learn to become the punisher? Like, okay, I'm going to exclusively make my reward off of punishing others and then others be like, no, I'm just going to collect my berries? Yeah, I think it was just for us not sharing the weights, just having individual agents, one neural network per agent was always the default for this line of work. And it didn't seem like there was any reason to change it here. In particular here, for modeling humans, who don't have the same policies as one another and things like that. Yeah. Yeah. And as an economist or a social scientist, or thinking about these tools, it always seemed like the shared weights just felt like assuming a can opener, right? It's just like assuming you're a way that key part of the problem, which is, you know, agent A has an incentive to free ride on the efforts of agent B. And we're trying to solve the problem of cooperation and coordination with individual agents. Coordination is much easier, right? If you make a small gradient change to your policy in a particular direction, but it's not just you, one agent, it's actually everyone makes that same change at the same moment. Then for certain problems, that can help coordination, not all problems. I doubt it made a huge difference in particular paper though. Yeah. So I did not find any specialization. So I don't think that they all that they develop different niches. But I do think it should be at least possible. So yeah, that's, I think, one of the reasons why we chose it. What would be main candidates to add here? I'm thinking of things like, in terms of abilities of these agents, if you wanted to go further, what would be questions, adjacent questions that you'd like to have answered from such a simulation and what would need to be added? Yeah, I'm thinking of things like maybe a bit of communication between the agents, some signaling, like I could like signal to others that I'm a good punisher or something like this or that. That's a question, and then we can go in a few directions. One thing that these are open is where do the norms come from, the content norms. Because here we just chose, this is a taboo area, this other one is a taboo area. But what we really want, if we want to have a model of cultural evolution, is a model where the norms themselves can emerge from the general training, the general learning of the agents. And so that is one direction that we started to go after this paper. We have another follow-up paper where we have a way for the content of the norms to evolve within the system. But it's also not perfect. It has continual learning problems, again, arise because if you have, you're kind of constantly changing the adaptive environment for everyone, and you can easily break reinforcement learning that way. So I think the next thing that's going to have to happen in this line, before it turns into like a real model of cultural evolution that feels like it can do the kinds of things we want cultural evolution models to do, is it will have to have some more effort on the continual learning side. Basically, make it so that the agents can kind of come up with one norm, so that society comes up with one norm, and then it can kind of change. So tipping point effects as it changes, because you see fads and trends and things. And none of that can really happen right now until we solve some continual learning issues. With respect to, you said something, we have to solve continual learning issues and so on. What is, like, I'm imagining there are quite a bunch of hyperparameters in this thing, not only reinforcement learning wise, like, what's my discount factor, blah, blah, blah, but also how many points do I give to what, right? I can give you gave four points per berry, like, well, that's the that's just a number. You give 35 points for for like punishing someone correctly. How sensitive are your findings to these to these things? Or how sensitive is the whole system to these parameters? So I think that's really hard to quantify, because a lot of the changes would be really meaningful, right, if you, let's say, make the berries so valuable that you never care about the poisoning, where you make the poisoning so weak that you don't have to worry about it. Any of these things you would expect to make a big difference because you've changed the balance of all the different things that you need to learn about. The thing that we tried that I thought was really encouraging was that we just reimplemented the whole environment and the agent and also tried a different type of learning agent on it and the results came out very similar. So that kind of made me pretty confident about like the overall observation that if you have this type of social learning problem where you learn from the observations of how others treat you, if you get more of those that helps. And that can be like a key component in like getting the overall population to the goal faster. How does one avoid like confirmation bias in these types of research? Because you probably have had some sort of idea of what you were going for and you know, like a hypothesis to show and like Occam's razor is kind of a brutal thing, right? And there is, if you see these results, you were like, oh yeah, this fits perfectly well with the hypothesis I had and so on. So what I'm not like I didn't not that I see anything wrong here, but I'm just wondering if you go into this with the hypothesis kind of what are the steps one needs to do to avoid sort of falling into confirmation bias? I mean, this kind of thing is about showing that a particular mechanism exists and is there. And what we don't know is of course, relative to all the other mechanisms that are supporting silly rules in the real world, how strong is this one versus other things? And we could talk about some of the other ones as well. And there's no way you could ever answer that from this kind of problem. I think though, and Rafael, you may want to say a little bit about this because it was you and our other co-authors that introduced this idea of testing individual agents at different points in training to say, can we confirm that that really is what the agents at these different stages are learning or have learned, right? That you know, because otherwise, you know, we're observing just this mess of eight agents interacting in this complex environment over and over again. I think that was really quite a great insight and innovation part of the innovation in the paper. And Rafael, you may want to say a little bit more about that because I think of that as the psych lab experiment for artificial agents in this context. Yeah. So I think you've touched upon this earlier. So one issue of course, is with all the metrics that you just get from the observations from the whole simulation is that it's not clear if you can take them at face value because there might be indirect effects that like... Please scroll up a little while he talks about this because we're thinking right above, yeah, right around there. So if you, for example, observe that they spend less time marked, is that because they get punished quicker or is it because they get marked less? And also, of course, the dependence of more being marked only creates the opportunity for being punished more, which then like creates pressure to get marked less. So because everything is entangled, it's really hard to know what do agents actually... What have they learned and how do they actually react to individual stimuli? What is it that they're actually trying to do? So the way we tried to approach this is similar to how psychology tries to approach it with humans that is like try to give them a controlled experiment, take them out of the complicated world, put them in like a lab where you just show them individual stimuli and see how they react. How quick are they to pick up the berry? That's what these pictures are. These are frames from that environment, this like test environment. Exactly. And then the results that we uncover are very similar to what you get from the observations. So sorry, from the metrics from the whole simulation. So that although this is a bit of a... Like there's some need to do generalization here. This is a bit different from the world that they actually inhabit. But even if you just show them one stimulus in isolation, they do start to just not pick up the berry that they have been punished for frequently. So it is like in that sense, like a very clear demonstration that they have learned the right thing even if the presentation of it is a bit different. But I'm not sure if it sort of answers your original question about the concept of... Yeah, that was my thing. I think it's more about... I think this is a big question for all modeling papers of like, what does it take for an economic model or a model of traffic or a model of how a disease spreads to be so good that you sort of trust it to make decisions based on it? I think that's sort of a long path that relies on many different papers sort of validating it. Calibration as well. I mean, ultimately, if you want to make real world predictions, real world decisions, you need to get real world data into the model. I think this is also something that comes from the collaboration between social scientists and computer scientists on this because we're seeing more and more computer scientists working on models that are interested in what's happening in the real world, like analyzing language models or multi-agent environments. And when you start bringing in social scientists who think about exactly this point, like, okay, so what's a good experimental design that allows me to reliably exclude alternative explanations for the phenomenon? And things like, and you should have a hypothesis before you start. You don't just run the simulation and say, hey, look at this cool stuff we discovered and report that. You try to craft something. We spent a lot of time on the experimental design on this one. And to exactly be able to respond to your potential critique of, well, how do we know you're not just giving us a just so story about what came out of this simulation? You said something like, to the effect of, we also think work like this is very, very important towards the direction of AGI. Do you want to explain a little bit what you meant by this? Because it is quite a different direction, AGI currently, that the biggest yee haw is in the direction of let's just make one language model really, really, really big. Where do you come from when you say work like this might be AGI material? Yeah, I'll start. We can all talk. So if you start from a place where what you want to do is make a human like AGI, and you can say to make a human like AGI, you need to capture all of the cognitive abilities that make human intelligence, perception, attention, memory, these kind of things. And you can have a single agent research program that does that. But from my perspective, and I think the scripture's perspective, that's not really what's important about human intelligence. It's not that we're better at perception or memory or attention or anything like that than other animals. That's not what's unique to us. It's not the secret of our success. It's a phrase that they always use in this space. But what is the things that are unique by humans are these more collective properties, things about how we cooperate, things about how we imitate each other, how our cultures evolve, and that's what you want to capture. So it's not the individual level social cognitive abilities. It's more like the group level social cognitive mechanisms, some of which might be ability like things like theory of mind, others might be more like representations, or some could even be like motivations. Like we talked about this intrinsic motivation to punish when you see a transgression, things like that. They're not exactly an ability, but in fact, they're not even things that we think of as terribly smart when you see an individual engaging in those kind of behaviors. At a group level, they might have a have a fact that influences our cooperation and how we learn from each other and how our norms work, how our institutions can be built and the way our technology develops and really contribute to all the things that we're proud of that come out of human intelligence. So if that's what human like intelligence is, then it follows that studying these kinds of issues is what we should be doing. And that's how I see this this line of work coming together in the AGI direction. And normativity in particular is a really important thing. I think it's not entirely just about like if you have a problem where that is a social dilemma or something, we need to cooperate. It's also just about kind of setting up the rules of the game that organize how we innovate, when we explore and when we don't. And norms like broadly construed so that they eventually include things like institutions that are really are critical for that. I think we kind of are that they set up the game that we're playing. We all work for companies and for universities. And these entities exist and structure our local incentives in ways that cause us to try to innovate. And I think that's really that's kind of that's how human intelligence as a group, collective intelligence works. It creates like local rules of the game for people to play so that intelligence can be applied in the right direction. So we can explore and do things. That's the that's that's where I come out with how I come out. Maybe we should all answer this question in different directions. Yeah, so I don't know if I have much to add to that. I think, yeah, the there's the perspective of developing intelligence from like cultural evolution of like populations of agents. And then of and then as Joel said, like norms are particularly interesting because they are if you have these multi agent systems, it's all about like the equilibria of how of that the behavior reaches. But the norms are the ones where you sort of take an active influence on the incentives of others. And that seems like it's a really important part of like a social structure. Let me add one thought here. When I get talks on this, I usually say, look, my favorite definition of of artificial intelligence is the capacity to act with foresight and appropriateness in a given set of circumstances. Well, that word appropriate in there is normativity. What in this environment? It's not just a matter of physics, right? Like what's there is notion of how you move a ball. But if you're going to interact with people in a meeting, if you're going to make decisions together, all of that is the structure that humans have invented. I think that's it's really critical to understand that that normative infrastructure is what allows us to accomplish so much collectively and to share information and learning across groups, across generations and to pay attention to the fact that that infrastructure needs to be generated and maintained by human behavior and perception. So I think this is to me, I say artificial general intelligence by definition has to include the capacity to participate and read this kind of normative information in the environment and participate in in in supporting it. So I don't know how we're going to generate artificial general intelligence without paying attention to normativity. So that's what we're I think that's the connection for me. I think the proponents of sort of the scaling hypothesis, they think that models can just pick it up out of reading stuff or so. If it's a static environment, right, but if this is dynamic, right? Your research investigates why things exist, why things come to be, why a mechanism might be there. Is there a prescriptive element to what you do? Would you dare say, well, what we figured out here, because of what we figured out here or over the course of our research, we can give recommendations to specific things in society of what we should do at some point. Like hey, how about a silly rule here? Is there something actually where you could say, here's a recommendation? I think so. Sorry, I'm on the recommendation side, I think. Yes, actually, this is a really critical point, and I worry about it a lot when we're thinking about alignment problems and so on. As we think about norms and values, there's this idea, if I asked you at the beginning, do you want to imbue your machine with just the important stuff or do you want to give it a bunch of silly stuff as well, silly rules to follow? Most people would answer that question, but clearly just the important stuff. We don't want the machines to be stupid like humans and worry about haircuts and Fuji and so on. But the point is that those silly rules are actually playing a very important role. In this model, they're helping to sustain those behaviors. In other work that we've done, we've shown how it contributes to robustness and the ability for the agents to read the state of the system, the enforcement system. Are the rules being enforced around here? Because if not, I'm leaving. I don't want to stay around and be vulnerable. I think a recommendation here is that actually you need some silly rules because there are cheap ways for agents to understand the state of the system. That's a critical thing to know to decide, do I continue to cooperate or do I go somewhere else? Is the scientific method just... This is no longer about RL, I guess. Is the scientific method kind of an antidote to silly rules? I figured at some point someone says, hey, I've actually tested it and we don't need to avoid the fish on Friday. It's actually not doing anything. I did my randomized controlled trial. Is this sort of like what percentage of silly rules that we have is impacted by this? More like 0.1%, 50%, 90%? Mostly don't. I think when we have a strongly held cultural belief like this, we don't give up in the face of evidence most of the time. So the scientific method maybe helps on the margins in some cases, but most of the time the silly rules overwhelm the evidence or we feel more strongly about adhering to the silly rule and enforcing it than we do about scientific method. And yeah, sorry. Not should, but I'm saying that's what people do. But there's some argument here that we are maintaining silly rules for a reason. That's the paper's about, of course. But it's not about any particular silly rule. And of course, if a silly rule becomes actually a harmful rule, then you really do want to have a mechanism for it. Where does the journey go from here for you? Like in this line of work? What are big, you've already mentioned a little bit like how do norms appear? What are other big unanswered questions that maybe other people who might want to get into this field might want to take a shot at? Another really interesting one that I don't know how we will get to, I hope you will mention, is how do you get systems of norms and then institutions? What's the relationship between norms and institutions? Can we have institutions emerge within our multi-agent systems? And what way would they really be different? Maybe like an institution has some kind of new personality to it or something like that. It doesn't matter who individuals are or something like that. But nothing like that has ever emerged in any institution we've run. But that would be really interesting to try. I think two of the things that I'm really interested in are thinking about robustness. And are groups that have developed these rule enforcement and compliance systems better able to respond to shocks and adapt to new information and changing environments? And then I think also to what extent does this become a more general mechanism for transfer learning across settings? Which is to say all I need to do when I go into a new environment and a group, particularly if it's already a stable group, is I need to look around and figure out what are these people think? What are you going to get punished for around here? What are you supposed to punish around here? And that can mean you learn a lot very, very quickly, which is how humans kind of work. If you got dropped down in the Arctic and you're lucky enough to land among the Inuit, the first thing you would do is say whatever those folks think is right or wrong to do, that's what I'm going to do. And fortunately, they'll be punishing you and throwing you out if you violate the rules. So you even have an added incentive to not think you can figure it out better than they can. So I'm interested in that, the idea that having this structure in place actually is part of what makes us so intelligent as we go down into new environments. Excellent. Is there anything else about this research that you want people to know? You want to shout out anything that is important you feel we didn't touch on? Well, one more thing. So this paper, along with all the other papers we've written recently, they generate both environments and agents, which we also packaged up together in an evaluation protocol on sewage environments that we've released, which is called Melting Pod. So it's anyone who wants to do multi-agent reinforcement learning research on environments that look vaguely like this, but on many different topics. Melting Pod is the place to go. We've put out a large number of different ones. We're putting out more all the time. It's a platform for doing multi-agent reinforcement research and having benchmarks you can compare to between algorithms and things. Cool. In this case, Rafael, Gillian, Joel, thank you so much for being here. I learned a lot. I hope to see you again soon.
 Why do social norms exist? And why are some of them really, really meaningful? And why do some of them make no sense at all? Like, why am I not allowed to wear this hat right here to a funeral? Okay, it might upset some people, but why? There is no benefit. There's no direct welfare impact to society with me wearing this hat.
 or not wearing this or wearing something else on my head. This is a question that we're going to investigate with today's paper. And yes, that has no inherent relationship with machine learning. But as you'll see, we can tackle this question or at least a part of the question we can give some evidence as to why these what's called silly rules might exist using machine learning, specifically deep reinforcement learning. So in this paper, people from different areas of expertise came together to say, why are some of these rules useful? And they said, why is
 Can we build a computational model of society?
 Can we build a little world of agents?
 Have them do some behavior, give them some rewards for certain things,
 and then we just observe what they do.
 And by observing, we can make some conclusions about,
 huh, this could be an explanation for a societal phenomenon that we see.
 So I like this paper because it's interdisciplinary.
 It uses deep reinforcement learning, specifically multi-agent reinforcement learning,
 in order to answer questions about society.
 And it is a little bit out of the box, which I like.
 So the video is structured.
 I first do a review of the paper by myself,
 and then I'm going to talk to the authors about the paper.
 This is one of the last videos where I recorded the interview before I did the review.
 But for this paper, it was actually super helpful because I'm a noob at this field.
 I don't know what I'm talking about when it comes to society and research in sociological questions.
 So it was very helpful to have the authors talk to me about the paper.
 But we don't just talk about the paper.
 We talk about many, many more things.
 And I highly invite you to watch the interview because it's really interesting.
 We talk about norms and societal systems of norms and hypotheses
 and what you have to pay attention to when you do research like this
 and what worked and what didn't and what it means.
 So please let me know if you like papers like this
 that are maybe a bit more distant from what we usually do.
 And if you do, then please let me know what other kinds of papers and what other areas exist
 where ML and specifically reinforcement learning or any kind of machine learning
 are used to investigate questions in other fields.
 All right, I'm going to leave it at that.
 And now I'll just do like a quick green screenshot
 because I know people are going to make emojis out of my face with this hat on.
 So.
 And that's that.
 Cheers.
 What they call silly rules.
 So the question is, our society has a bunch of norms of what you should do and shouldn't do.
 And these norms are known by the people and they are enforced by the people.
 You're being shamed if you don't follow the norms.
 A lot of those norms are really good, like wash your hands after you use the toilet.
 But there are a lot of norms that are also just arbitrary.
 Like what kind of hairstyle is good and bad or acceptable or not acceptable.
 What words are rude and things like this.
 And these are called silly rules.
 And the question is, why do these exist?
 Now, this is not a question of machine learning.
 However, this paper applies deep reinforcement learning
 in order to give some evidence to why these rules can exist.
 So I like the mixture here of sort of using reinforcement learning as a tool
 to investigate these mechanisms by using a computational model.
 You can break down a lot of things.
 Usually, if this were a psychology paper, people would go into a lab,
 they would recruit people, and then they would try to design an experiment around these norms and so on.
 And that's cool and all.
 But if you use a computational model, you can answer different questions.
 You can control for different variables and so on.
 So it's very attractive to use reinforcement learning for that.
 So we're going to look at what this paper says right here.
 Not as much into the RL part because that is fairly straightforward.
 But just what it does and what it says.
 And I'd like just to show you maybe a little bit because I thought it was pretty cool
 that this is yet another application of machine learning and specifically reinforcement learning
 that enables progress in a different field.
 So I hope you enjoy this.
 Yeah, they introduce the paper by saying there are a lot of norms.
 Something that differentiates human from other animal society is this presence of norms.
 And some of many of these norms, say, generate direct benefits for individual and group well-being,
 like, you know, reciprocity, sharing of rewards, what you should eat,
 what you shouldn't eat, and so on.
 Very often, these rules have some sort of a benefit to society.
 They say, but, however, the normative landscape is also populated by many norms
 that appear essentially arbitrary and without direct material consequences.
 And we're not necessarily fighting about this.
 Like, people can always say, well, but this rule may have some use.
 But let's just, for now, let's assume that there exist norms that really could be different,
 and it would make not a difference in total welfare, or at least a direct difference, right?
 The paper here argues that there is an indirect difference.
 The paper argues that by introducing these silly rules, the indirect benefits are that
 agents learn the enforcement behavior of the rules more clearly.
 And therefore are better at enforcing the important rules.
 But we'll get to that in just a second.
 So here are some of the examples of silly rules that they mention.
 Men are expected to wear pants, not skirts, which in some societies is the case,
 and others isn't, right?
 There are words or hand gestures that should not be used in polite company.
 There are rules about how one's style of hair or what one wears on one's head, and so on.
 So they call these silly rules.
 Silly rules means essentially a norm that is in society, is very, you know, taken seriously,
 but is essentially arbitrary.
 They say they're meaningful and enforced, but they have no direct first order impact on welfare.
 So why do they exist?
 There are some hypotheses.
 They list some here.
 They say, for example, silly rules may remain stable by virtue of their incorporation into
 larger normative systems that also include important rules, which essentially means that
 the silly rules, they make sense if they are part of a bigger system that also contains
 the important, which means the useful rules.
 And so the hypothesis here is that the addition of the silly rules into a society somehow
 helps the society to comply more broadly or more or more or better or more accurately
 with the important rules.
 So the addition might be some might be a benefit in the total benefit, like total setup of the system.
 In this paper, they say we describe a mechanism through which silly rules can benefit a society.
 Our argument is based on the dynamics of learning in a group that lacks a priori knowledge
 of which of the rules are truly important.
 So there is a group, there's a society, there are a bunch of norms already present,
 and a priori, no one can tell which ones of those are important and which ones aren't,
 because if they could tell, they could just say, well, that one is not important,
 which is what's happening kind of with the scientific method, right?
 We know that some things aren't as important and with time, people stop doing them.
 But initially, you know, there's no way of knowing.
 And that's what they investigate.
 It's important that they say, they describe a mechanism, right?
 They don't necessarily say this is how society works, right?
 Because society is way more complex, but they do describe one possibility, one mechanism,
 one reason why these silly rules could exist.
 And they show that this mechanism, if you implement this in a mini-society,
 will lead to a total welfare benefit.
 Their explanation is the following.
 The skills involved in third-party norm enforcement readily transfer from norm to norm,
 while the skills involved in compliance are norm to norm.
 The skills involved in compliance are norm-specific.
 What that means is, essentially for every norm, you have to learn how to follow that norm.
 So these are the skills involved in compliance.
 They are norm-specific.
 If, you know, there's a food I shouldn't eat, then I have to learn to avoid that food.
 And then if there is some sort of like a way, like, please share if you have enough,
 like that's a norm, I have to learn how to do that.
 For many norms, the skills to behave in accordance to the norm are very specific to the norm.
 However, the enforcement, this enforcement skills, they transfer from norm to norm.
 So what's the enforcement skill?
 For example, shaming someone if they don't follow a norm.
 That's very, that's similar from norm to norm, whether they don't follow the hygiene norms
 or the interaction norms or the food norms or the hairstyle norms is always the same
 to shame someone into compliance or to, I don't know, deduct from their social credit score
 or something like this.
 So they argue that the skill of enforcing norms transfer while the skills of following norms
 don't transfer as much.
 And therefore, they say, the silly rule may provide greater opportunity to practice
 third party norm enforcement.
 And through that, the third parties will also become better at enforcing the true, the useful
 norms.
 So the addition of silly rules might simply make it easier for people to learn to shame
 others into submission.
 And by that, they will be more effective at shaming them when it comes to the good norms,
 which obviously they don't know.
 So they're just going to shame for all the norms.
 But overall, it is positive in welfare.
 So what they do is they have this environment right here.
 You can see the environment right here.
 So up on up here is a schematic of the environment, but this is kind of the representation.
 They are going to have a map, which is a 2D map.
 You can see that right here.
 That's the map.
 And sorry, on this map, you have agents.
 So an agent right here, that's sort of a little person that's walking around.
 The person can walk around so they can walk up left, right, and so on.
 Every person sees a little window around themselves.
 They see what's happening around.
 There are sort of obstacles there, but there are also these berries.
 And the berries, I don't know if you can see them on the screen, but the berries, this is
 a berry.
 These are two berries right here.
 They come in different colors.
 So the agent's goal is to move around and collect these berries.
 Every berry they get, they get some sort of points.
 You know, they collect them.
 That's the reward.
 There are enough berries so that there is no meaningful competition between agents.
 There is one other thing they can do, and that's zap someone.
 They call it even zapping.
 So in this case, I'm going to guess something like this agent right here is zapping this
 agent down here.
 And the yellow thing is a punishing, punishing beam.
 Essentially, that just means that the agent can zap another agent, which will cause the
 zapping agent to lose a bunch of points and the zapped agent also to lose more points.
 The only addition now comes with the poison berries.
 So sometimes some of the berries are poisoned and there will be a color selected for which
 berry is poisoned.
 For example, let's call all the green berries here.
 They're poisoned when an agent picks up a poison berry.
 They are they they won't see necessary.
 They won't see it themselves, but they will be poisoned.
 And after they pick up a poison berry, 100 steps later, they will start to lose health
 or I think they will just they will not gain as much from eating other berries.
 That's it.
 So there is a very delayed, very slow punishment for eating poisoned berries that takes the
 agent a long time to learn that.
 However, if now if you get zapped while you're poisoned, that gives the zapper a benefit.
 So let's call this person Alice here and this person Bob.
 If Alice zaps Bob and Bob is fine, then Alice loses some points and Bob loses some points.
 However, if Bob is poisoned, then Alice gains a bunch of points for zapping Bob.
 So Bob is poisoned, loses points and Alice gains points by zapping Bob.
 I do think so.
 The zapping cures Bob, I think.
 So one zap will actually cure Bob, but Bob loses a lot of a lot of points.
 Hey, y'all, it's Yannick from the future.
 I made a small mistake right here in that I claim that zapping cures the poison, which
 it does not.
 The idea is that zapping removes the mark.
 So when a player eats a poisoned berry in this normal rule condition, they become marked
 and zapping cures the mark.
 If you zap a marked player, you get points, but zapping removes the mark.
 It does not cure the poison.
 The poison is still active.
 The idea is obviously that the players learn to avoid the poison in the first place because
 they don't want to get marked because they don't want to get zapped.
 And now in the silly rule condition, also a second berry activates the mark, but that's
 not a poisoned berry.
 And this you would expect that it's more noisy and therefore learning is more difficult.
 But it turns out under the silly rule condition, learning is actually more efficient.
 And that's kind of the point of the paper.
 So again, the zapping doesn't cure the poison.
 It just removes the mark in whatever way that mark happens to be on the map.
 Happens to be on the player in the first place.
 Back to the video.
 Yeah, there's one last thing and that you can see here in the marking.
 So when an agent is poisoned, so when they after they've eaten a poisoned berry, they
 become marked, which means that all the other players will see that they are poisoned.
 Now, this is the setup.
 What you can pretty quickly see.
 So no rules is here.
 We have berries and we have poisoned berries that give you a delayed punishment.
 Then this is what I just described with what's called the important rule condition, which
 is that if you eat a poisoned berry, you become marked.
 And then if a third party and other players sees that they can zap you and they gain a
 bunch of points.
 So you can see that pretty quickly.
 What is going to happen is that the agents, they learn to eat berries, but then pretty
 quickly they learn to spot the marked agents and they zap them.
 And then after that also very quickly, the other agents will learn to avoid the green
 berries because they realize wait, every time I get a green berry, I get zapped later.
 And that's how that's how the agents avoid learn to avoid the green berry.
 Note, we have to clarify some things.
 This paper isn't about how the norm of not eating the green berries comes to be because
 obviously that's kind of like God given right here.
 The marking is done by the environment.
 The rewards are clearly set up such that people learn to avoid the green berries.
 That's not the issue right here.
 The question that the paper has is how quickly can the agents learn to enforce that norm?
 So how quickly do they catch on zapping others?
 Right?
 And what is the overall welfare?
 So the norm itself is set by the environment or by the designers of the experiment.
 We are not trying to learn to avoid the green berries.
 We are trying to learn to avoid the green berries through the effect of poison.
 But we simply directly give rewards for zapping the marked agents.
 And that means we...
 Deus ex machina...
 Ex nihilo...
 What means just like we command a norm onto the system and we see how the agents react.
 So that is obviously what's happening here is not a secret.
 Imagine that by the way the agents they use an actor critic.
 They use a simple conv net and an actor critic framework to learn right here.
 What I find interesting is that there are 12 neural networks.
 So the system keeps 12 neural networks that are initialized with the same weights,
 but they're different neural networks.
 And 8 of the 12, I'm gonna just select three or four right here,
 but imagine that's 8 of 12.
 8 of the 12 are then each episode drawn to compete in the ring.
 They compete for a thousand time steps, then they get their learning updates,
 they get put back and then for the next thing 8 others are drawn.
 Which I found pretty interesting.
 It's a way to sort of get diversity into the system.
 Now what does that have to do with silly rules?
 So far we've built up an environment.
 We forced a norm onto it by giving reward for punishing these marked agents.
 And we've discovered that agents learn pretty quickly to enforce that norm,
 which in turn makes all the agents avoid the poison berries
 as a consequence of being punished by the norm.
 Now we introduce this silly rule.
 So the silly rule means that there are poisoned berries, which are these ones,
 but there are also other berries that we will call taboo berries.
 The taboo berries, they're just fine.
 They're just, you know, they're fine.
 They're healthy.
 You can eat them.
 You get a bunch of points for eating them.
 That's fine.
 However, if you eat the taboo berries, you will also become marked,
 just like the poison berry eater.
 Right?
 So these are indistinguishable markings.
 And therefore, the agents that learn to gain points by zapping the taboo berries
 will also gain points by zapping the ones that ate the taboo berries.
 What's even worse is that they also get reward for zapping the taboo berry eaters.
 So there's no difference in the reward for zapping that you get
 if you zap a poison berry eater or a taboo berry eater.
 You just, whenever you zap a marked player, you get some points.
 Again, it's not about how the agents learn to avoid the poison berries.
 It's how they react to given norms.
 Right?
 So again, we enforce the norm of you should eat neither the poison berry nor the taboo berry.
 Of course, the agents don't know which one is the poisonous one.
 They just know they get zapped after eating either the pink or the green berry.
 So how does that go?
 That's sort of the question of this paper.
 We've introduced a silly rule, which on a surface serves no purpose.
 The green, making the green berry taboo serves no purpose other than it's just,
 it's just a rule and you get punished for not following it.
 It even decreases the overall welfare a little bit because now you don't want to eat the
 green berries anymore, which means that you don't get as many points.
 The question is, can the introduction of the silly rule get you an overall reward?
 An overall benefit as a society?
 That's the question.
 So we'll go on a little bit.
 They say our model allows us to separate the learning of enforcement and compliance
 behaviors from the learning of the norm content itself.
 That's what I repeatedly emphasized because I had a lot of trouble when reading this paper
 to really get this.
 They don't want to, they don't want to, they say here, we designed an experiment in which
 norm content was fixed in advance by the experimenter, namely which berries are taboo.
 The question is, how do they react to it?
 So this is a brief recap.
 If a player breaks the taboo, they change color in the observation of other agents
 viewing their transgression.
 They become marked.
 If a player is marked, other players can collect a reward by punishing them.
 This creates an incentive for players to learn to punish rule violations and thus for players
 to learn not to violate the rules.
 And these are the results.
 We show that individuals achieve higher overall welfare in a world where eating the poison
 berry is taboo.
 That's condition one.
 This is clear.
 This is logical.
 We take a delayed punishment for eating poison and we essentially bring it to the present
 by having people zap the poison people and them learning to avoid it.
 However, the main results, sorry, they say even with the cost of enforcement, overall
 group welfare is higher with the norm than without.
 We then show our main result that the value of the normative order is higher if the set
 of norms in this regime includes not only important rules such as the rule against eating
 poisonous berries, but also silly rules which make the eating of a harmless berry taboo
 and bring about the same third party.
 Punishment.
 So they show there is a situation right in which you can gain by introducing such silly
 rules because enforcement skills are learned faster.
 Let's just quickly look at the agent architecture.
 If you're into machine learning or RL or so, this should be rather familiar to you.
 So the agent, they see raw pixels up here.
 There's a neural network.
 It's a CNN followed by an MLP.
 There is an actor critic.
 So there is a value function and there is a policy function.
 Actor critic, very basic actor critic algorithm.
 This is obviously a very easy environment for reinforcement learning and that makes
 it ideal to use multi agent RL here to gain some insights.
 As I said, we have 12 agents, 8 out of 12 play in 64 environments in parallel.
 And they get the replay buffers and they update those weights.
 All right.
 Yeah, I've mentioned these things.
 I've mentioned these things.
 Now let's look at the results.
 So first of all, let's look at fraction of time spent poisoned.
 Like how?
 So here is time step strain.
 So this is over the course of training.
 Right.
 So what fraction of the time do the agents spend?
 Does an average agent spend poisoned?
 If there is no rule, you can see that there is a constant fraction of the time agents
 spend poisoned.
 Essentially over the course of this training, they don't learn really to avoid the poison
 berries and therefore, yeah, because the reward is just too delayed.
 I guess the RL algorithm also isn't too powerful, but you can see that there is a clear difference
 between the important rule and the silly rule.
 So important rule means there is only one rule, shouldn't eat the poison berries and
 silly rules that means that there is in addition this silly rule.
 So the agents here quickly, they spend less total time poisoned.
 And the question is, is why?
 So let's look at some other effects that the introduction of the silly rules have.
 Total taboo berries eaten.
 You can see that at the beginning, about double the amount of taboo berries are eaten
 under the silly rule than under the just important rule, which makes sense because twice as many
 berries are taboo.
 So you'd eat twice as many of them in the same time.
 But you can see that there is a crossover.
 This decreases and there's actually a crossover.
 So after a while, less taboo berries are eaten than in the important rule setting, even though
 there are more taboo berries, right?
 So somehow these agents learn faster to avoid the taboo berries.
 Total punishments.
 Now, obviously, again, at the beginning, there are double as many taboo berries, so double
 as many marked players.
 So they go, the number of punishments goes up pretty quickly.
 And then there's a crossover point where after a while, there is less punishment going on
 than in the important rule.
 So these societies, they learn faster.
 And that's, I think, the point.
 You can see that at the end, there's often sort of the same result, the same outcome,
 but in this intermediate stage.
 And remember, society is always in flux, kind of.
 So one can argue that very often we are at all times in sort of this intermediate stage.
 So in this intermediate stage, it's actually an overall benefit.
 Fraction of time spent marked goes down as well pretty quickly, obviously, because people
 are more marked.
 And collective return.
 So here is the actual result.
 If you have no rule at all, collective return goes up at the beginning, it's actually the
 highest, but then flat lines, right?
 Because people keep getting poisoned and that hurts.
 If you, however, use this important rule thing, then at the beginning, it's not as great,
 because if you punish, the rewards are structured such that if you punish, you decrease the
 total welfare.
 Even though you as an agent gain some points, the total number of points in society decreases
 as a result of punishment.
 So you can't just punish more and more and more and expect to get more and more.
 You have to expect the collective return to grow.
 So yet still, because agents learn to avoid the poison berries through punishment.
 So at the beginning, there's lots of punishment.
 That's why the reward, the collective return is lower, but then they learn.
 And as they learn, they learn to avoid the poison berries, then they don't need to punish
 as much anymore, right?
 And then the reward goes higher than if you had no rule at all.
 Most interestingly, however, in the case of the addition of the silly rule, you can see
 that at the beginning, there is a decrease in collective return as people punish around,
 like they punish each other to death.
 Yet, yet, very quickly, this goes up and actually becomes the highest collective return there
 is.
 And you can see in this intermediate period right here, there is clear benefit to having
 these silly rules around because the society is much quicker and much better at learning
 to avoid the poison berries because, because, and you can see from the time series right
 here, because they learn much more quickly to punish, to punish people who eat the wrong
 berries, not only the poison, but also the silly ones.
 And because they're much quicker at punishing, the agents have more opportunity to learn
 to avoid these berries, and that's what gives you the higher return.
 They do investigate what these agents have learned.
 They say psychology experiments with human participants address the issue of learning
 what people have learned individually by isolating specific mechanism and testing in these controlled
 conditions, such as reactions to particular stimuli.
 They want to do the same thing computationally.
 So they take these agents from their training run, they put them in inference mode, and
 they give them like a little environment like this.
 So they start apart from the berry and the episode ends on contact with the berry.
 So then there you can give them a berry and see if they eat it or if they don't eat it.
 So if you have no rule at all, if you don't have this marking rule or anything like this,
 here again, it's time steps trained, but remember, we don't train the agent on this task, we
 train it on the original task, then at certain checkpoints, we take it out, we put it in
 little lab and we see what happens.
 Also, the y axis here is inverted.
 So 30 is down here, which means 30 time steps.
 If the line is here, it means the agent has not eaten the berry.
 If the line is up here, or like somewhere up here, it means the agent has immediately
 eaten the berry.
 You can see that if you have no rule, agents, they just eat the berry.
 Doesn't matter if it's poisonous or not, right?
 The pink is poisonous.
 It makes a little bit of a difference, but not really.
 They just eat it.
 If you add the important rule, they quickly learn to avoid the poison berry.
 You can see that right here.
 If you add the silly rule, they also learn to avoid not only the poison berries, but
 also the taboo berries.
 They also, in fact, learn to avoid the healthy berries a little bit more, but this comes
 back over time.
 There is a bit of an unlearning right here, and I do ask that in the interview.
 They specifically highlight...
 So these are different berries.
 Now, just isolating the times when they give the agent a poisoned berry, you can see that
 the reaction to the poisoned berry is much, much bigger if you are in the condition that
 contains the silly rule compared to if you're in the condition that doesn't contain the
 silly rule in this intermediate regime right here.
 And also, you know, the punishing is way quicker.
 So they measure how long it takes you to punish.
 It's way quicker when you have the silly rule.
 So that's essentially the evidence that they say, look, these agents, they learn the skill
 of punishing.
 They learn the skill of running after someone who is marked and therefore punishing them.
 And that gives the agents the opportunity to learn to avoid poisoned or marked berries
 altogether.
 And because there is more punishment, because the agents are better at punishing more early
 on, they learn to more quickly avoid the poisoned berries.
 So the overall argument again is that the skills of punishing are transferable between
 tasks and the addition of a silly rule, even though it brings some negative welfare because
 it's a rule you need to follow, like you incur some cost, it could still be total benefit
 overall because the introduction of the rule just trains people in punishing others for
 not following the rules and therefore trains people in following rules and therefore trains
 people in following the important rules.
 Remember, in this society, people have don't know, the assumption is they don't know which
 of the rules are beneficial and which ones aren't.
 So these were in the discussion now, they say from the perspective of an agent learning
 the skills necessary to effectively enforce their society's norms, the additional violations
 constitute additional opportunity for practice, and thus promote a faster rate of improvement
 in their command of the mechanisms, sorry, of the mechanics of third party punishment.
 Now obviously, this doesn't go forever, right?
 You can't just add silly rules until you know, like until the world is just made of rules
 and expect well, we're always going to have much higher welfare.
 But there is a regime where that is the case, and we might as well live in that regime in
 our societies.
 They say enforcement and compliance are asymmetric in the sense that the former is a skill that
 may be applied without modification to any norm that's enforcement.
 Since many of the sub behaviors involved in third party punishment are directed towards
 the violator, for example, chasing them, not towards the event of the violation itself.
 Thus, they are transferable skills generically applicable to any norm.
 And yes, I get it if you say, for example, avoiding food is also transferable and so
 on.
 Sure, sure.
 But I think this sentence here that a lot of punishment behaviors are directed towards
 the violator and not towards the event of the violation itself, that it makes sense
 that these skills are more transferable.
 The interpretation of our key result is that the role of silly rules in human normative
 systems may in part be to help train a society's ability to comply with important rules.
 And that is the result.
 The paper goes into more detail, obviously, in all of these results in the setup in why
 it's important and so on.
 But I'll leave it at that for now.
 I hope you gain some insights into how reinforcement learning can help other fields to get some
 insights by modeling sort of these computational little societies and just introducing aspects
 of the real world.
 And then just seeing how that pans out.
 It wasn't clear at all from the beginning that the introduction of the silly rule here
 would bring this improvement in sort of the intermediate timeframes.
 And that's just really interesting.
 And it's kind of a different way of approaching the questions of why does silly rules exist
 in society.
 Questions like these, it's a different way of approaching them than just putting some
 humans in a lab, which has its own problems, right?
 So I think this just gathers some evidence and it's pretty cool.
 And it's an opportunity for interdisciplinary research, which I like.
 And I hope this was fun to you as well.
 And I'll see you around.
 Bye bye.
 Hello everyone.
 Today I have with me here three of the authors of the paper about spurious normativity enhances
 learning of compliance and enforcement behavior in artificial agents, Gillian Hadfield, Joel
 Liebow and Rafael Custer.
 You are an assembly of people with way different backgrounds that have somehow come together
 and focused on a very cool intersection between machine learning and social sciences.
 Welcome to the channel and yeah, welcome.
 Thanks for having us.
 Great to be here.
 So I mean, the first things first, in machine learning, we've had these trends of just making
 like clickbaity titles.
 I feel your field should pick that up because a title like this, it's like that is an instant
 desk reject.
 You got to have like a little acronym, like spell or something, like just four letters
 or so and then, or a question.
 But yeah, it's a pretty cool.
 Yeah, it is.
 We did have a somewhat more intriguing title than the journal told us to change.
 Yeah, we did have silly rules in the title for this reason and they were nervous about
 that.
 Okay.
 There is still some veneer of professionalism in other fields of science, not in ours.
 Yeah, I was very, very happy to see this paper because it connects something that I know
 to something that I don't know.
 And I think, you know, us machine learners were sort of always in the same areas.
 And this goes a little bit outside of my comfort zone.
 So I thought it was pretty cool.
 How did you get like the idea of writing something like this, of connecting these fields?
 Like where does it come from?
 I can start with how I came to it.
 So my background is in computational neuroscience.
 That's where I did my PhD in.
 And when I came to DeepMind, I was thinking about how do we build artificial general intelligence
 and reading lots of things about human intelligence and realized that intelligence isn't really
 in the brain.
 So my whole PhD on neuroscience was maybe not as helpful as I thought it would be.
 But intelligence is actually a collective phenomenon that is more supported by how societies
 work and how we cooperate with each other and learn from each other and things like
 that.
 And so since then, I've been trying to build human like AGI in a way that is more like
 trying to make a society of AGI.
 And this was one piece of work that came out of that after meeting Jillian.
 Maybe Jillian can speak.
 Yeah, maybe I can say a little bit.
 So I'm a social scientist.
 I don't build these systems.
 I think about and study how human normative systems work.
 Right.
 Those are our systems of norms and our systems of rules.
 And I'm very interested in that from a systemic point of view.
 What are the attributes of the systems that make them stable and adaptive and contribute
 to human progress and evolution?
 And so I've been thinking about working on those kind of models, these economic modeling
 tools.
 And Joel's team at DeepMind had produced some papers studying some very standard problems
 in the economics literature on like tragedy of the commons and showing how they could
 use sort of those multi-agent reinforcement learning setups to study tragedy of the commons,
 which is sort of econ 101.
 I saw those papers, got very excited and said, oh, but we could really dramatically increase
 the sort of the social science component of this work.
 And I had been working with Dylan Hadfield-Minell, who's also on this paper on this concept of
 silly rules.
 And so actually, I think I tracked you down, Joel, and started a conversation a number
 of years ago.
 And we gave a talk.
 Yeah.
 We spoke afterwards.
 Yes, right.
 Oh, that's right.
 I came and gave a talk at DeepMind.
 And yeah, so I was very excited to be connecting up these two worlds.
 And then you needed someone to actually do the work.
 And then that's where Rafaela came in.
 I think I don't have much to add to Joel's story.
 So my background is also in cognitive neuroscience and psychology.
 And I work on topics that are sort of on the intersection of decision making and memory
 in humans and in AI.
 So social cognition, as well as learning from others or how groups behave is similar.
 And also questions of behavioral economics are all sort of all in the scope of what I'm
 really interested in.
 So I think this is, yeah, like a good example of where these things come together.
 Yeah, it's pretty cool.
 So to give the brief introduction to maybe the paper, I think it's maybe for the machine
 learners it's valuable to start with this one right here.
 So we have this environment.
 There are different agents inside of it.
 I think you already always have eight agents that take part in an episode.
 The episode can go up to like a thousand steps.
 In each step, each agent has the ability to move around.
 The goal is to collect the berries.
 It has like a little window view around itself of the world.
 And there is one other action.
 It can like zap someone else, right?
 It can zap, punish an agent.
 And we'll get to that in a bit.
 So these berries that are around, you deliberately made the berries plentiful.
 So there's no issue of like, yeah, competition or anything like this.
 There are three conditions that you compare and these are kind of your experimental conditions.
 Do you want to maybe say like, if you gave the pitch about your own method, I think this
 kind of is the core right here.
 How would you describe it?
 I might want to say what the purpose was.
 Yeah, sure.
 Experimental conditions, right?
 From my perspective, one thing that I think following on from what Jillian said a minute
 ago, it's true.
 We really did have a bunch of papers that were kind of reproducing economics 101 kind
 of ideas about a tragedy of the commons and things like that.
 And we had a sequence of those papers.
 And this was the first time we were really trying to like contribute back and say something
 actually new.
 That's not just like a new way of coming to the same kind of results that people already
 had in economics for centuries.
 And so this particular area we're trying to connect with is a field that's interested
 in cultural evolution and cumulative culture and things like human uniqueness.
 They see humans as an ultra social species.
 It's like critical to the niche that we are in.
 It requires a it's a cultural niche.
 We learn from each other.
 That's how our technologies work, how our societies are put together.
 And that's what's what makes us different from other primates.
 And so within that literature, one thing that's interesting is how is how we cooperate.
 And social norms are one kind of mechanism of cooperation.
 There's others like reciprocity and things like that.
 And then within that field, there's another question of like, we have all kinds of social
 norms, some of which seem to be relevant to cooperation, and some of which just seem to
 be irrelevant things.
 Like we can have a we can moralize all kinds of behaviors like you're supposed to wear
 clothes and you're not supposed to wear a hat in this circumstance or whatever.
 And the question that is like, well, social norms are so important for cooperation.
 Why are there all these other social norms that are like, just not doing that?
 I mean, is you have this concept of the you have this concept of the of the silly rule,
 right, which is a fantastic name.
 And it describes sort of a norm that isn't directly valuable to anything that that considers
 like group fitness or even personal fitness.
 Yet, does this actually exist?
 Like is there a rule where we can conclusively say this is a silly rule and not, you know,
 we might be missing some hidden advantage?
 Well, that's the point.
 You can never say that for any rule, really.
 Because you're inside the system, you never know whether this is there for some important
 reason or not.
 But I think this is a key thing is sort of just to sort of place this work in the context
 of the work that gets done on trying to explain human rules and norms.
 And so we have people come at this mostly from a functional point of view, like it's
 a solution to a game theory.
 It's a solution to a coordination challenge, or it's a solution to like a hot dove type
 problem where we're going to waste resources fighting over something that or cooperation,
 like Joel was saying, right?
 So most of our work in social science has come at the question of explaining norms by
 saying they serve this functional purpose.
 But it seems very clear we have lots and lots of rules where you could say, look, nothing
 would be different from a functional point of view.
 If we said you wear bright stripes at a funeral instead of black, or that you stand this far
 apart rather than this far apart.
 It's just once you start noticing silly rules defined in this way as no direct impact on
 welfare.
 Only impact, which is what we're showing, is the role those silly rules play in helping
 to stabilize a system by which people can enforce the important rules.
 So I think that's a key thing.
 So it sort of starts as a puzzle.
 Here's this thing that seems to be true of every human society you look at.
 Food rules, right?
 What we eat and don't eat is often a good example.
 Very tons across different groups and communities over time.
 Why do we have them?
 Why are they stable?
 There's really no good explanations in literature.
 So we got really interested in thinking about the role they play in supporting what I'd
 call the normative infrastructure, which is what you draw into enforcing important rules.
 If you're going to punish people for stealing your stuff or punish people for going back
 on their contracts, you need to have coordinated and incentivized your community to enforce
 rules.
 And what we're looking at is what's the role of silly rules in helping to create that structure.
 It is a bit like the value of just having rules.
 And if you have more rules, then you'll be better at following rules and people will
 be better at enforcing rules.
 And it's just like more rules sort of lead to...
 Because rules are a transferable skill.
 It's the enforcement part.
 And that's what you would want to get at right here.
 So your goal is sort of if we train agents and if we introduce like a silly rule like
 this, this skill would sort of transfer to beneficial rules whenever we actually have
 beneficial rules.
 So in the first context here, there are berries and there are poisonous berries.
 If you eat the poisonous berries, some when later, you'll kind of die, but your reward
 will shrink from eating new berries.
 So it will be like a very delayed thing.
 And in this case, we all know reinforcement learning isn't really good at super long rewards.
 You also have a discount factor, right?
 So the long rewards don't even matter.
 I could even imagine if a berry is close to me and I knew it was poisoned, I'd be like,
 meh, right?
 It's a hundred steps away.
 Who cares, right?
 I'll just eat it and I'll go back.
 But let's assume the agents actually want to avoid that.
 And then you have a silly rule and an important rule.
 The silly rule being you can mark or the rules are you can mark agents, right?
 Agents are marked.
 If you eat a berry that is taboo, you get marked.
 So you change the color and the perception of the others.
 So you yourself don't see it, but you change color in the view of the other agents.
 And if you are marked, other agents can collect the reward if they punish you.
 And so what we're doing with these three different conditions is we're sort of fixing what the
 norms are.
 That's the sort of the experiment is if you set the norms, what are the effects downstream
 on the ability of the agents to learn to enforce those norms and to then comply with the underlying
 rules that they are representing.
 And in the important rule condition, the taboo berry actually coincides with the one that
 is poisonous.
 So that's a really important rule for your group to have that should, if everybody learns
 to follow it, lead to everybody avoiding getting poisoned.
 In the silly rule condition, you still have the important rule.
 But on top of that, you also get marked for eating a berry that is fine and doesn't actually
 poison you.
 So there's the potential for twice the amount of transgressions and then also punishment
 behavior following that.
 The important thing is you get marked just the same.
 So in the third condition, whether you eat a poison berry or the berry that's fine, but
 just marked as taboo, you get marked the same.
 So there's no distinction.
 And the others collect a reward, whether you're poisoned or not, it's enough that you are
 marked right.
 So that that is how you sort of set these norms in place.
 Because I was I was sort of like, okay, the agents I have to figure out which one's poisoned,
 like no, they do get a reward as soon as soon as they zap someone who is marked.
 And now we're going to see what happens in a little bit as a result of these experimental
 conditions.
 But my question first is a motivation to punish those who have transgressed normative code
 and you want to like those those ones, they violated it, we want to enforce on them our
 social ethic or whatever.
 The question is a little bit.
 So there is this is like a microcosm, right?
 Sorry, there's a cat right here.
 This is a microcosm system.
 And I you know, there's always this in economics, there's always that the micro economists versus
 the macro economists, right?
 They and they and they kind of fight because the micro economists, they come up with their
 models and their simulations and their formulas.
 And then the macro economists are like, well, if you actually look at the whole world, it's
 completely different, right?
 Maybe you can get some insights, right?
 But there's always this danger of, you know, this enclosed system with these very constrained
 things.
 As soon as you introduce something else, it might just change the entire game.
 Is this something that you're, you're kind of avoiding somehow or worried about or not
 worried about?
 Should I take that one as the economist in the in the crowd?
 So I think there's there's a way in which what we're doing is the same kind of thing
 that micro economists which I am are doing, which is looking at, you know, idealized or
 schematic settings and doing theory about that in order to gain insight and generate
 testable predictions.
 And you're not trying to say this is a map of the world exactly as it is it's saying
 we can gain insight into what would be the impact of changing that price or that cost
 or increasing competition, that kind of thing.
 And so I think what we're what we're doing here is and we refer to this as kind of micro
 foundations, which actually lots of macro economists are interested in micro foundations,
 which is, is can we do a simulation like this to solve a problem that we can't do closed
 form with our theoretical tools like we would normally do like, you know, solve for an equilibrium
 or solve for, you know, a solution to a game theoretic problem.
 This is allowing us to solve a much more complex problem and gain insight and then demonstrate
 this, you know, we've got this hypothesis that said our agents will learn faster and
 better to both enforce and then therefore comply with rules if there's a silly rule
 in the environment.
 So I think a bit is kind of similar methodologically to that.
 I think it's got this this relationship to cultural evolution, not exactly one to one.
 We don't think humans started off like only being able to recognize pixels in the world,
 but that the idea that this is something that evolves over time, but we're not trying to
 kind of model like evolutionary game theory tries to in some ways model what would happen
 with repeat populations over time.
 So that's how I think about it.
 Well, I think it pays that we now jump to the results a little bit to take it ahead
 before we discuss sort of the like broader implications or anything like this.
 So is it fair?
 Like correct me if I'm wrong.
 I would characterize your main result or your main thing you derive from it that if I impose
 the taboo on the poison berry through this mechanism of agents getting reward, zapping
 each other, the population will sort of learn to avoid the poison berries better if then
 if if they just get the delayed anti reward.
 In addition, if I now also introduce another taboo berry, that's fine.
 It's silly rule, right?
 The agents can collect even more reward by by zapping, you would say they are learning
 the skill of enforcing rules, which is a generalizable skill.
 And through by becoming better at enforcing rules, they're sort of faster catching on
 to the fact that, you know, I should punish people for eating the wrong things.
 Therefore, the whole population learns to not eat these types of berries faster.
 Is that about in the ballpark?
 Yeah, there's there's an evolution of like the skills or what has been learned.
 Like at first, the agents need to learn to even perceive the world and then effectively
 eat berries that then increases to them actually getting poisoned a lot because they eat the
 wrong very a lot.
 And once that is in place, and you actually have a lot of marked agents, then it is possible
 to learn about the punishment and that it's that you can collect a reward for punishing
 marked agents.
 Once that is in place, then you have the opportunity to actually learn to avoid the berry you want
 to avoid because you are avoiding the punishment.
 But for that, you need all of the other agents to have learned to actually discourage this
 behavior.
 So this is sort of the nice progression of that one skill relies on another skill having
 been learned beforehand.
 And the silly rule helps exactly in providing more observations and more training for that
 learning of skills.
 And this is the sort of result you could only get with a model that is really focused on
 learning of skills.
 Another thing, another aspect of it is there's a very long temporal credit assignment problem,
 which is very difficult for reinforcement learning in the case where there's just poison
 berry.
 But in the case where they're being punished for eating that berry, then you're moving
 closer in time the negative thing to the event.
 So it's much easier to learn about it.
 This evolution you mentioned is visible in the graphs, right?
 So you first have like the total the total taboo berries eaten, it kind of goes up at
 the beginning because you get a reward for eating berries, then people learn to punish
 others, right?
 So that in time, you see that spike after the other spike.
 And then the like various things happen like the fraction of time spent poisoned and the
 fraction of time spent marked, they go down dramatically as a consequence of the punishments
 increasing.
 And at the end, sort of the collective return goes beyond what you would just have.
 So the difference here, I guess, is the credit assignment problem difference.
 There doesn't seem to be too much of a difference in the end result.
 Like if you let the game play out between the just the good rule, let's say and the
 silly rule.
 What is like so your claims are more about the evolution of the thing and somewhere in
 the middle, there might be an advantage to having the silly rule.
 Is that?
 Yeah, I was gonna say I think that's that's what's emphasizing that it's about learning
 these behaviors of, you know, the relationship between what you eat and Oh my god, somebody
 showed up and that's me.
 Right, learning that and then learning Oh, I get this reward if I zap somebody who is
 marked.
 So learning those behaviors, you know, once they're once they're learned in a stable,
 stable way, then the benefit of the silly rule is kind of okay, we've accomplished our
 learning objective.
 My own intuition is that that that the silly rules are going to help you with robustness
 so that when the environment changes, right, and they got to learn something new so that
 even though in our environment, it they they converges at the end, my guess is you can
 then introduce kind of the shock of you know, the rain didn't come this year or a different
 we're in a new part of the world and there's a different dangerous berry.
 Then then so I think that's that that that's likely if you sort of did follow on these
 experimental results, you have some more you draw this conclusion that what is the common
 thing is sort of the mechanism of enforcing rules.
 The agents they they learn this, this is a transferable skill.
 And by having sort of more taboos around, they learn this faster.
 What is different?
 Like what differentiates this hypothesis from the hypothesis that agents are better at avoiding
 some color of berry because by introducing, you know, a new taboo berry, I teach the agents
 that you know, this new berry is also taboo.
 And I say with the same argumentation that it may be not the enforcement that they learn
 in common, it may be avoiding some color of berry.
 Well, that's sort of the consequence, right?
 That's the compliance part.
 Yeah.
 From there, they can't see anything different until someone has enforced something on them.
 Because if they need a berry that is taboo, they're marked only in the eyes of others,
 they can't see themselves.
 And for the silly rule, nothing happens at all.
 It's just that they ate the berry and it became marked in everyone else's eyes.
 But from that perspective, nothing happened at all.
 So there's there's no effect on them in any way until the punishment comes first.
 Okay.
 Yeah, that's the only way that they could ever learn to comply.
 Is there a...
 And that's one of the nice the graphs in there to Rafael, the sort of showing that it is
 that sequence of learning to punish and then learning to avoid getting getting poisoned.
 A social equivalent to getting a reward for punishing someone who has transgressed a taboo.
 If I think to myself, the progression of this would be it would be more like if I enforce
 some taboo, then long term that will lead to more group welfare because everyone keeps
 to the rule, we eat less poisoned berries or we follow rules in general.
 And there is an aspect of group fitness that also reflects on me.
 You chose to directly give me reward if I punish someone for transgressing.
 Is this purely just because you wanted to like hard code these norms?
 Or is there like a social equivalent to that?
 Yeah, I'll take that from one perspective.
 And then I think we can do it from a few different ones here because this has multiple kind of
 ways of thinking about it.
 So the one you can see it as an intrinsic motivation agents just are motivated intrinsically
 to punish the transgressions of their norm that they have.
 So it's like some kind of like righteous anger on the part of the agent that just saw this
 this transgression.
 And then they're motivated to punish it.
 And that's a very kind of natural human emotion that we all feel for different norms.
 Like we could have totally totally different norms in mind, we can from different cultures
 to different places, but we might still feel a feel some like this is a transgression that
 we've just witnessed.
 I think it's whatever it is.
 That's one interpretation we could have.
 We have several others.
 There's this interesting one about medieval Iceland, maybe someone could say.
 Yeah, let me let me jump in there.
 So so so the fact that humans have this capacity for that they have this practice of third
 party punishment.
 So that's that really is distinctive about humans in the evolution of species.
 And it's a great puzzle.
 Why do humans spend resources punishing people for, you know, doing, you know, committing
 harm to others?
 It's that third party piece.
 And so we've got people in, say, behavioral economics who think it's about altruistic
 punishment.
 That's a little bit of what what the way I understand what Joel was talking about with
 intrinsic motivation that you just have a taste for punishings.
 We got a whole bunch of in behavioral economists who study sort of like, you know, people willing
 to pay money to be able to punish people for hurting other people.
 But it's a real it's a real puzzle in the story of cultural evolution about where that
 comes from.
 And so we have people who are in second order, like we have we have punishment for people
 who fail to punish.
 So we do actually have critiques that say, hey, how come you didn't say anything when
 that person said that harassing thing to the other person around the meeting table?
 Right.
 We have reactions to people who don't respond and don't punish people for violating our
 contract rules.
 And and in this anyway, it's a real, real puzzle.
 And we're hard coding it here.
 Some evolutionary anthropologists model it as a trait of punishment, like we have punishers
 and non punishers.
 My own view is that that's actually that that's the fundamental behavior to try and explain
 why do we end up with humans willing to spend personal resources punishing on somebody else's
 behalf, because that's the secret of our success.
 I was species.
 And should we do the medieval Iceland example?
 That's what that one's.
 Oh, oh, many of the lights.
 Yes.
 Right.
 So don't refer to the fact that I sort of been around looking at it really is about
 decentralized punishment.
 So the key thing to know about medieval Iceland is they had lots and lots of rules and they
 had no enforcers, no public enforcers, no police, no soldiers, no chiefs who had any
 power.
 They just have one individual, the law speaker who was responsible for reciting all the
 rules every year at a big gathering and who was the person you can go and ask, is this
 allowed?
 Not allowed.
 And that coordinates everybody on being willing.
 And they had very clear, not only rules, but what you could do, but also the penalties.
 Like if you did this, you had to give up 10 sheets.
 If you did that, you got kicked off the island.
 And what you need to do is coordinate your community to actually implement that punishment.
 And that's what they did really very effectively with zero public enforcement apparatus.
 Now eventually it becomes more efficient to have some enforcement apparatus, but individuals
 enforcing the rules is a really big part of both human history and even today really important.
 Think about mask mandates.
 Think about our pandemic rules.
 We're relying very heavily on community enforcement and non-enforcement.
 So the conclusion, the general conclusion is introducing a silly rule sort of makes
 group welfare higher or achieves the welfare faster, let's say by mechanism of, I learn
 a transferable skill and so on.
 So adding one silly rule, good.
 Adding two silly rules, adding three, adding four, like at some point, there must be a
 detriment to having only silly rules.
 How far would this go out?
 Is one the optimum?
 Is there some optimum of silly rules?
 Is this known?
 Can you assess that maybe with your simulation?
 So we haven't specifically tested this, but I think your intuition is right that there
 would be an optimal number because also every rule introduces costly effects because overall
 someone punishing someone else, overall destroys reward.
 So you end up with a net negative.
 So the more punishment there is, it's overall worse for the group.
 So the benefit needs to be quite large to overcome all of this additional punishment.
 So I think it would depend on how hard is, so first of all, how costly are they?
 If they're very cheap, then you can get away with more.
 The other thing is how hard is the thing that you're trying to learn?
 If it's very difficult to learn the punishment behavior and you need lots and lots of additional
 observations to do so, then I think additional rules would help.
 Whereas if it's very easy to learn, then you barely need any additional observations and
 you're just stuck with the bill.
 So I think it depends on that.
 I think it's some sort of inverted U shape with some optimal amount.
 I see in these graphs a little bit that sometimes at the end, actually trends reverse a little
 bit, especially in the silly rule case.
 And I've seen it here and here.
 It's also prominent in these sort of single agent tests which you do, which I really like.
 You take a single agent, you put it in a controlled environment.
 It's not training, it's just at some point during training, it's like an eval set.
 But also here, you kind of see these sort of reverse trends as training progresses.
 What happens there?
 Are they becoming really good?
 Do they learn the actual reward of being poisoned?
 Or what's going on there?
 Do they learn to avoid the punishers?
 I suspect that what happened there is some amount of unlearning because if you are very
 effective at teaching the population to not get marked and they effectively avoid all
 the taboos, then this behavior just doesn't occur anymore.
 You will just forget that you've ever learned that.
 So I think if this were to keep running, they might have to at some point relearn it.
 But then the question is if they actually would relearn it because now they have competition
 from different things.
 Maybe they're very good at collecting berries now, so maybe they're not as interested anymore
 as even learning about the punishment dynamics at all because the counterweight of their
 other behaviors is different.
 So I think this turns into a continual learning problem if you just let it run for a very
 long time.
 There's a covariate shift when the behavior of marked agents existing and then being available
 to punish is very different.
 Your structure has a bit of a special thing in it which I found, which is that you have
 12 different agents, let's say 12 different neural networks that you train.
 In every episode, you choose eight of them to compete, whereas sometimes or a lot of
 times in multi-agent reinforcement learning, I have like one neural network, maybe with
 a bit of randomness, but essentially every of the multi-agents has the same weights.
 Let's say they're all shared.
 Was there a particular reason why you chose this specifically?
 Not only having different neural networks for each agent, but also to always sort of
 select subsets of them.
 And also, the follow-up is have you discovered that they diverge?
 I would be interested, did one learn to become the punisher?
 Like, okay, I'm going to exclusively make my reward off of punishing others and then
 others be like, no, I'm just going to collect my berries?
 Yeah, I think it was just for us not sharing the weights, just having individual agents,
 one neural network per agent was always the default for this line of work.
 And it didn't seem like there was any reason to change it here.
 In particular here, for modeling humans, who don't have the same policies as one another
 and things like that.
 Yeah.
 Yeah.
 And as an economist or a social scientist, or thinking about these tools, it always seemed
 like the shared weights just felt like assuming a can opener, right?
 It's just like assuming you're a way that key part of the problem, which is, you know,
 agent A has an incentive to free ride on the efforts of agent B. And we're trying to solve
 the problem of cooperation and coordination with individual agents.
 Coordination is much easier, right?
 If you make a small gradient change to your policy in a particular direction, but it's
 not just you, one agent, it's actually everyone makes that same change at the same moment.
 Then for certain problems, that can help coordination, not all problems.
 I doubt it made a huge difference in particular paper though.
 Yeah.
 So I did not find any specialization.
 So I don't think that they all that they develop different niches.
 But I do think it should be at least possible.
 So yeah, that's, I think, one of the reasons why we chose it.
 What would be main candidates to add here?
 I'm thinking of things like, in terms of abilities of these agents, if you wanted to go further,
 what would be questions, adjacent questions that you'd like to have answered from such
 a simulation and what would need to be added?
 Yeah, I'm thinking of things like maybe a bit of communication between the agents, some
 signaling, like I could like signal to others that I'm a good punisher or something like
 this or that.
 That's a question, and then we can go in a few directions.
 One thing that these are open is where do the norms come from, the content norms.
 Because here we just chose, this is a taboo area, this other one is a taboo area.
 But what we really want, if we want to have a model of cultural evolution, is a model
 where the norms themselves can emerge from the general training, the general learning
 of the agents.
 And so that is one direction that we started to go after this paper.
 We have another follow-up paper where we have a way for the content of the norms to evolve
 within the system.
 But it's also not perfect.
 It has continual learning problems, again, arise because if you have, you're kind of
 constantly changing the adaptive environment for everyone, and you can easily break reinforcement
 learning that way.
 So I think the next thing that's going to have to happen in this line, before it turns
 into like a real model of cultural evolution that feels like it can do the kinds of things
 we want cultural evolution models to do, is it will have to have some more effort on the
 continual learning side.
 Basically, make it so that the agents can kind of come up with one norm, so that society
 comes up with one norm, and then it can kind of change.
 So tipping point effects as it changes, because you see fads and trends and things.
 And none of that can really happen right now until we solve some continual learning issues.
 With respect to, you said something, we have to solve continual learning issues and so
 on.
 What is, like, I'm imagining there are quite a bunch of hyperparameters in this thing,
 not only reinforcement learning wise, like, what's my discount factor, blah, blah, blah,
 but also how many points do I give to what, right?
 I can give you gave four points per berry, like, well, that's the that's just a number.
 You give 35 points for for like punishing someone correctly.
 How sensitive are your findings to these to these things?
 Or how sensitive is the whole system to these parameters?
 So I think that's really hard to quantify, because a lot of the changes would be really
 meaningful, right, if you, let's say, make the berries so valuable that you never care
 about the poisoning, where you make the poisoning so weak that you don't have to worry about
 it.
 Any of these things you would expect to make a big difference because you've changed the
 balance of all the different things that you need to learn about.
 The thing that we tried that I thought was really encouraging was that we just reimplemented
 the whole environment and the agent and also tried a different type of learning agent on
 it and the results came out very similar.
 So that kind of made me pretty confident about like the overall observation that if you have
 this type of social learning problem where you learn from the observations of how others
 treat you, if you get more of those that helps.
 And that can be like a key component in like getting the overall population to the goal
 faster.
 How does one avoid like confirmation bias in these types of research?
 Because you probably have had some sort of idea of what you were going for and you know,
 like a hypothesis to show and like Occam's razor is kind of a brutal thing, right?
 And there is, if you see these results, you were like, oh yeah, this fits perfectly well
 with the hypothesis I had and so on.
 So what I'm not like I didn't not that I see anything wrong here, but I'm just wondering
 if you go into this with the hypothesis kind of what are the steps one needs to do to avoid
 sort of falling into confirmation bias?
 I mean, this kind of thing is about showing that a particular mechanism exists and is
 there.
 And what we don't know is of course, relative to all the other mechanisms that are supporting
 silly rules in the real world, how strong is this one versus other things?
 And we could talk about some of the other ones as well.
 And there's no way you could ever answer that from this kind of problem.
 I think though, and Rafael, you may want to say a little bit about this because it was
 you and our other co-authors that introduced this idea of testing individual agents at
 different points in training to say, can we confirm that that really is what the agents
 at these different stages are learning or have learned, right?
 That you know, because otherwise, you know, we're observing just this mess of eight agents
 interacting in this complex environment over and over again.
 I think that was really quite a great insight and innovation part of the innovation in the
 paper.
 And Rafael, you may want to say a little bit more about that because I think of that as
 the psych lab experiment for artificial agents in this context.
 Yeah.
 So I think you've touched upon this earlier.
 So one issue of course, is with all the metrics that you just get from the observations from
 the whole simulation is that it's not clear if you can take them at face value because
 there might be indirect effects that like...
 Please scroll up a little while he talks about this because we're thinking right above, yeah,
 right around there.
 So if you, for example, observe that they spend less time marked, is that because they
 get punished quicker or is it because they get marked less?
 And also, of course, the dependence of more being marked only creates the opportunity
 for being punished more, which then like creates pressure to get marked less.
 So because everything is entangled, it's really hard to know what do agents actually...
 What have they learned and how do they actually react to individual stimuli?
 What is it that they're actually trying to do?
 So the way we tried to approach this is similar to how psychology tries to approach it with
 humans that is like try to give them a controlled experiment, take them out of the complicated
 world, put them in like a lab where you just show them individual stimuli and see how they
 react.
 How quick are they to pick up the berry?
 That's what these pictures are.
 These are frames from that environment, this like test environment.
 Exactly.
 And then the results that we uncover are very similar to what you get from the observations.
 So sorry, from the metrics from the whole simulation.
 So that although this is a bit of a...
 Like there's some need to do generalization here.
 This is a bit different from the world that they actually inhabit.
 But even if you just show them one stimulus in isolation, they do start to just not pick
 up the berry that they have been punished for frequently.
 So it is like in that sense, like a very clear demonstration that they have learned the right
 thing even if the presentation of it is a bit different.
 But I'm not sure if it sort of answers your original question about the concept of...
 Yeah, that was my thing.
 I think it's more about...
 I think this is a big question for all modeling papers of like, what does it take for an economic
 model or a model of traffic or a model of how a disease spreads to be so good that you
 sort of trust it to make decisions based on it?
 I think that's sort of a long path that relies on many different papers sort of validating
 it.
 Calibration as well.
 I mean, ultimately, if you want to make real world predictions, real world decisions, you
 need to get real world data into the model.
 I think this is also something that comes from the collaboration between social scientists
 and computer scientists on this because we're seeing more and more computer scientists working
 on models that are interested in what's happening in the real world, like analyzing language
 models or multi-agent environments.
 And when you start bringing in social scientists who think about exactly this point, like,
 okay, so what's a good experimental design that allows me to reliably exclude alternative
 explanations for the phenomenon?
 And things like, and you should have a hypothesis before you start.
 You don't just run the simulation and say, hey, look at this cool stuff we discovered
 and report that.
 You try to craft something.
 We spent a lot of time on the experimental design on this one.
 And to exactly be able to respond to your potential critique of, well, how do we know
 you're not just giving us a just so story about what came out of this simulation?
 You said something like, to the effect of, we also think work like this is very, very
 important towards the direction of AGI.
 Do you want to explain a little bit what you meant by this?
 Because it is quite a different direction, AGI currently, that the biggest yee haw is
 in the direction of let's just make one language model really, really, really big.
 Where do you come from when you say work like this might be AGI material?
 Yeah, I'll start.
 We can all talk.
 So if you start from a place where what you want to do is make a human like AGI, and you
 can say to make a human like AGI, you need to capture all of the cognitive abilities
 that make human intelligence, perception, attention, memory, these kind of things.
 And you can have a single agent research program that does that.
 But from my perspective, and I think the scripture's perspective, that's not really what's important
 about human intelligence.
 It's not that we're better at perception or memory or attention or anything like that
 than other animals.
 That's not what's unique to us.
 It's not the secret of our success.
 It's a phrase that they always use in this space.
 But what is the things that are unique by humans are these more collective properties,
 things about how we cooperate, things about how we imitate each other, how our cultures
 evolve, and that's what you want to capture.
 So it's not the individual level social cognitive abilities.
 It's more like the group level social cognitive mechanisms, some of which might be ability
 like things like theory of mind, others might be more like representations, or some could
 even be like motivations.
 Like we talked about this intrinsic motivation to punish when you see a transgression, things
 like that.
 They're not exactly an ability, but in fact, they're not even things that we think of as
 terribly smart when you see an individual engaging in those kind of behaviors.
 At a group level, they might have a have a fact that influences our cooperation and how
 we learn from each other and how our norms work, how our institutions can be built and
 the way our technology develops and really contribute to all the things that we're proud
 of that come out of human intelligence.
 So if that's what human like intelligence is, then it follows that studying these kinds
 of issues is what we should be doing.
 And that's how I see this this line of work coming together in the AGI direction.
 And normativity in particular is a really important thing.
 I think it's not entirely just about like if you have a problem where that is a social
 dilemma or something, we need to cooperate.
 It's also just about kind of setting up the rules of the game that organize how we innovate,
 when we explore and when we don't.
 And norms like broadly construed so that they eventually include things like institutions
 that are really are critical for that.
 I think we kind of are that they set up the game that we're playing.
 We all work for companies and for universities.
 And these entities exist and structure our local incentives in ways that cause us to
 try to innovate.
 And I think that's really that's kind of that's how human intelligence as a group,
 collective intelligence works.
 It creates like local rules of the game for people to play so that intelligence can be
 applied in the right direction.
 So we can explore and do things.
 That's the that's that's where I come out with how I come out.
 Maybe we should all answer this question in different directions.
 Yeah, so I don't know if I have much to add to that.
 I think, yeah, the there's the perspective of developing intelligence from like cultural
 evolution of like populations of agents.
 And then of and then as Joel said, like norms are particularly interesting because they
 are if you have these multi agent systems, it's all about like the equilibria of how
 of that the behavior reaches.
 But the norms are the ones where you sort of take an active influence on the incentives
 of others.
 And that seems like it's a really important part of like a social structure.
 Let me add one thought here.
 When I get talks on this, I usually say, look, my favorite definition of of artificial intelligence
 is the capacity to act with foresight and appropriateness in a given set of circumstances.
 Well, that word appropriate in there is normativity.
 What in this environment?
 It's not just a matter of physics, right?
 Like what's there is notion of how you move a ball.
 But if you're going to interact with people in a meeting, if you're going to make decisions
 together, all of that is the structure that humans have invented.
 I think that's it's really critical to understand that that normative infrastructure is what
 allows us to accomplish so much collectively and to share information and learning across
 groups, across generations and to pay attention to the fact that that infrastructure needs
 to be generated and maintained by human behavior and perception.
 So I think this is to me, I say artificial general intelligence by definition has to
 include the capacity to participate and read this kind of normative information in the
 environment and participate in in in supporting it.
 So I don't know how we're going to generate artificial general intelligence without paying
 attention to normativity.
 So that's what we're I think that's the connection for me.
 I think the proponents of sort of the scaling hypothesis, they think that models can just
 pick it up out of reading stuff or so.
 If it's a static environment, right, but if this is dynamic, right?
 Your research investigates why things exist, why things come to be, why a mechanism might
 be there.
 Is there a prescriptive element to what you do?
 Would you dare say, well, what we figured out here, because of what we figured out here
 or over the course of our research, we can give recommendations to specific things in
 society of what we should do at some point.
 Like hey, how about a silly rule here?
 Is there something actually where you could say, here's a recommendation?
 I think so.
 Sorry, I'm on the recommendation side, I think.
 Yes, actually, this is a really critical point, and I worry about it a lot when we're thinking
 about alignment problems and so on.
 As we think about norms and values, there's this idea, if I asked you at the beginning,
 do you want to imbue your machine with just the important stuff or do you want to give
 it a bunch of silly stuff as well, silly rules to follow?
 Most people would answer that question, but clearly just the important stuff.
 We don't want the machines to be stupid like humans and worry about haircuts and Fuji and
 so on.
 But the point is that those silly rules are actually playing a very important role.
 In this model, they're helping to sustain those behaviors.
 In other work that we've done, we've shown how it contributes to robustness and the ability
 for the agents to read the state of the system, the enforcement system.
 Are the rules being enforced around here?
 Because if not, I'm leaving.
 I don't want to stay around and be vulnerable.
 I think a recommendation here is that actually you need some silly rules because there are
 cheap ways for agents to understand the state of the system.
 That's a critical thing to know to decide, do I continue to cooperate or do I go somewhere
 else?
 Is the scientific method just...
 This is no longer about RL, I guess.
 Is the scientific method kind of an antidote to silly rules?
 I figured at some point someone says, hey, I've actually tested it and we don't need
 to avoid the fish on Friday.
 It's actually not doing anything.
 I did my randomized controlled trial.
 Is this sort of like what percentage of silly rules that we have is impacted by this?
 More like 0.1%, 50%, 90%?
 Mostly don't.
 I think when we have a strongly held cultural belief like this, we don't give up in the
 face of evidence most of the time.
 So the scientific method maybe helps on the margins in some cases, but most of the time
 the silly rules overwhelm the evidence or we feel more strongly about adhering to the
 silly rule and enforcing it than we do about scientific method.
 And yeah, sorry.
 Not should, but I'm saying that's what people do.
 But there's some argument here that we are maintaining silly rules for a reason.
 That's the paper's about, of course.
 But it's not about any particular silly rule.
 And of course, if a silly rule becomes actually a harmful rule, then you really do want to
 have a mechanism for it.
 Where does the journey go from here for you?
 Like in this line of work?
 What are big, you've already mentioned a little bit like how do norms appear?
 What are other big unanswered questions that maybe other people who might want to get into
 this field might want to take a shot at?
 Another really interesting one that I don't know how we will get to, I hope you will mention,
 is how do you get systems of norms and then institutions?
 What's the relationship between norms and institutions?
 Can we have institutions emerge within our multi-agent systems?
 And what way would they really be different?
 Maybe like an institution has some kind of new personality to it or something like that.
 It doesn't matter who individuals are or something like that.
 But nothing like that has ever emerged in any institution we've run.
 But that would be really interesting to try.
 I think two of the things that I'm really interested in are thinking about robustness.
 And are groups that have developed these rule enforcement and compliance systems better
 able to respond to shocks and adapt to new information and changing environments?
 And then I think also to what extent does this become a more general mechanism for transfer
 learning across settings?
 Which is to say all I need to do when I go into a new environment and a group, particularly
 if it's already a stable group, is I need to look around and figure out what are these
 people think?
 What are you going to get punished for around here?
 What are you supposed to punish around here?
 And that can mean you learn a lot very, very quickly, which is how humans kind of work.
 If you got dropped down in the Arctic and you're lucky enough to land among the Inuit,
 the first thing you would do is say whatever those folks think is right or wrong to do,
 that's what I'm going to do.
 And fortunately, they'll be punishing you and throwing you out if you violate the rules.
 So you even have an added incentive to not think you can figure it out better than they
 can.
 So I'm interested in that, the idea that having this structure in place actually is part of
 what makes us so intelligent as we go down into new environments.
 Excellent.
 Is there anything else about this research that you want people to know?
 You want to shout out anything that is important you feel we didn't touch on?
 Well, one more thing.
 So this paper, along with all the other papers we've written recently, they generate both
 environments and agents, which we also packaged up together in an evaluation protocol on sewage
 environments that we've released, which is called Melting Pod.
 So it's anyone who wants to do multi-agent reinforcement learning research on environments
 that look vaguely like this, but on many different topics.
 Melting Pod is the place to go.
 We've put out a large number of different ones.
 We're putting out more all the time.
 It's a platform for doing multi-agent reinforcement research and having benchmarks you can compare
 to between algorithms and things.
 Cool.
 In this case, Rafael, Gillian, Joel, thank you so much for being here.
 I learned a lot.
 I hope to see you again soon.