The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies (Paper Explained)
deep learning reinforcement learning society gini index welfare taxes brackets progressive regressive us poor rich equality redistribution outer loop world resources labor trade neural networks ppo
Hail the AI Tax Collector! This very visual framework has RL Agents maximize their coins in a tiny world through collecting, building and trading. But at the same time, the government is also an AI trying to maximize social welfare via taxes. What emerges is very interesting.

Paper: https://arxiv.org/abs/2004.13332
Blog: https://blog.einstein.ai/the-ai-economist/

Abstract:
Tackling real-world socio-economic challenges requires designing and testing economic policies. However, this is hard in practice, due to a lack of appropriate (micro-level) economic data and limited opportunity to experiment. In this work, we train social planners that discover tax policies in dynamic economies that can effectively trade-off economic equality and productivity. We propose a two-level deep reinforcement learning approach to learn dynamic tax policies, based on economic simulations in which both agents and a government learn and adapt. Our data-driven approach does not make use of economic modeling assumptions, and learns from observational data alone. We make four main contributions. First, we present an economic simulation environment that features competitive pressures and market dynamics. We validate the simulation by showing that baseline tax systems perform in a way that is consistent with economic theory, including in regard to learned agent behaviors and specializations. Second, we show that AI-driven tax policies improve the trade-off between equality and productivity by 16% over baseline policies, including the prominent Saez tax framework. Third, we showcase several emergent features: AI-driven tax policies are qualitatively different from baselines, setting a higher top tax rate and higher net subsidies for low incomes. Moreover, AI-driven tax policies perform strongly in the face of emergent tax-gaming strategies learned by AI agents. Lastly, AI-driven tax policies are also effective when used in experiments with human participants. In experiments conducted on MTurk, an AI tax policy provides an equality-productivity trade-off that is similar to that provided by the Saez framework along with higher inverse-income weighted social welfare.

Authors: Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C. Parkes, Richard Socher

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Alright, today we're going to find out why AI is much better at governing people, why poor people really should pay more taxes, and how Donald Trump is just a normal human. Alright, we'll dive into it. We're looking at the AI Economist by Salesforce Research. Now Salesforce Research has kind of created a simulated world environment where they can place agents in it and the agents, they can move around, they can collect resources, they can trade those resources, and they can use those resources to build houses and that will earn them coins. And each agent wants to maximize its own coins, but also there's the government and the government can set taxes. So they collect money from everyone and they redistribute it. And the goal now is going to be that the AI handles both the agent and the taxes and we want to maximize the social welfare of the entire population. Alright, that's the goal. So the paper here is called The AI Economist Improving Equality and Productivity with AI Driven Tax Policies by Stefan Cheng and Alexander Trott and other people from Salesforce Research and Harvard University. So as I said, this is a simulated environment and the simulated environment works like this. There is a 2D plane, kind of like a game playing field and in this game there are agents. Here you can see the agents, there are always four agents. Where? Oh, down here. What are you doing in the corner? Come on, be productive. The agents are in this world and they can do certain things. They have certain actions at their disposal. So first of all, they can move around. They can move down, left, right and so on. Whenever they walk past a resource tile, they collect the resource. This is stone and this is wood. So there are two kinds of resources. And then the last actions the agents have is building a house. One wood and one stone will create one house and the house gives you coins. So this is a house and that will give you coins. But how much coins you get is different from agent to agent and this represents the agents different skill levels. This is an abstraction and the kind of economic theory behind it is that the income inequality in people, one of the main drivers of it is that they are skilled differently and therefore are able to convert one unit of labor into more money than another lower skilled worker. So this is here represented by the fact that maybe if this agent here builds the house, they'll get 50 coins. But if this agent here would build the same house, they'll only get 10 coins. So we'll call this here a high skilled worker and this here a low skilled worker. Now the last thing, sorry, I saw the last thing before, but the very last thing the agents can do is they can trade. So if one agent has too many resources and the other one has not enough, they can trade those resources among each other for those coins. So once you build a house, you collect some coins, you can then either go and collect more resources or you can use those coins in order to buy resources off of other people. This is unlucky. No coins, no houses, and no resources. Look at them. Oh yeah, so you also can't move across the water here. You can only move on the grass. You can also not move through a house, which gives you some interesting abilities because you can just build a house right here. And yeah, so and you can't move over other players. But these are the rules are pretty simple. And the goal here is for the agents to maximize the number of coins they get in 1000 steps. So the number H here is 1000, which is the number of steps that the agents can take before the game is over and it restarts again. So each agent is using reinforcement learning in order to learn how to achieve the maximum number of coins. Now the policy is of course going to be different depending on whether that is a high or a low skilled worker. The catch here is that outside of this there is the government, the government here, let's draw this big house with the flag of our fictitious nation, which is like this. That's the flag. And the government will observe what's happening here and they will issue a tax taxes. So it will issue a tax distribution. Now how do you imagine that? So if you imagine the government says something like this for the first 10 coins you own, you owe us 5% of that. For the next 10 coins, so from 10 to 20 you earn, you owe us 10% and so on. So if you earn even more, you owe us more and more percent of those extra coins. This is what you might know as a progressive tax schedule. The more you earn, the more percentage wise you pay on that extra earned money. This is what you might be used to, but there are other tax schedules and the exact histogram you see or the exact how many percent for which amount of coins, that is the action of the government. So the government decides on the taxes and the taxes are just collected from the income. So if an agent earns these coins, then it has to pay taxes to the government and the government will redistribute all the taxes it has collected equally among the population. So if you pay a lot, you might lose through this process and if you just pay a little taxes you might gain through this process. So that's it. That is the basic premise of the game. The agents are using reinforcement learning and I believe the newness of this paper is also that the government now is using reinforcement learning in order to determine the optimal tax policy. There is kind of this inner loop here and there is this outer game where the government also tries to maximize the RL. And what does the government try to maximize? Good question. It is a measure that's called social welfare. Now social welfare consists of two things and they have this here way down in the paper. Social welfare in this paper consists of two things. First of all, economic productivity, which basically just means how many coins has anyone produced. It doesn't matter who, but just the total amount of coins produced. The second one is income equality and this is related to the Gini index. So if you plot the cumulative distribution of wealth, a fully equal society would be a straight line because 50% of the people would have 50% of the money and so on. But almost all true societies have something like this where 50% of the people might have 10% of the money and the rest 50% of the people has the other 90%. And the measure of inequality is this area here. This is called the Gini index and 1 minus this area is what this paper has as an equality measure. So the higher this number, the more equal is the society in terms of their income distribution. Now what is actually optimized for here is this thing, equality times productivity. So you want both to be high, your income equality and your productivity. There's a trade off here of course, but you can have multiple ways to trade that off and that will give you the different thing. They call this the social welfare function. And that's the thing that the government RL agent optimizes for. So you can see here already the free market, even though it's the most productive, produces the most coins because if you have a free market means no taxes. If you have no taxes, then people are basically encouraged to earn more money because they don't have to pay taxes on them. As soon as you tax them, they're less encouraged to earn more money. And therefore if you have no taxes, the most coins will be earned in total. But the equality suffers. So the equality is the lowest among these things considered. If you compare that to the AI economist, the AI economist achieves the highest social welfare. It achieves the highest equality, but it doesn't suffer as much in productivity as other systems here. And the baseline systems are first of all, the US federal system. This is not particularly tied to the US. This is basically every system or most of the systems that you have currently in the world is the progressive tax system and the SAES formula, which I believe is an economically theory based system, which is a regressive tax schedule. You can see them down here where the US federal will be progressive, means the more you earn, the more percentage wise you pay. While the SAES formula will be regressive, which generally means the more you earn, the less you pay. I believe this was derived under some assumptions to be the optimal tax distribution. And the AI economist will come to this in a second. Let's actually just look at one of these things first, one of these games, how this plays out. The cool thing here is that they have pretty flashy animations. So you can look how does one of these games turn out. Now this is a free market game and you can see the agents moving around collecting things, building houses. And you might notice that one of the agents, namely agent one, is just building all of the houses. And generally just kind of being a dick, being in everyone's face and kind of building things everywhere. And the other ones don't. Or just very few, like the light blue on the bottom left builds some houses. On the right you can see how the distribution of wealth is structured. And you see agent one ends up with most of the wealth. Now the size of the circle I think is the total productivity. So you can see this grows over time mainly because agent one becomes so rich. And if you analyze this, if you analyze what's happening here, then you'll see that agent one and I might be... Yeah, they have a graph up here. So it is very interesting what happens. This is kind of the same game. So agent one here is this orange dot and agents two, three and four are these dots here. And this graph here is coin from trading. So how much money they win or lose from trading. Now the green bars are trading wood and the brown bars are trading stone. So you see agent number four, which is the lowest skilled, the skill is just determined at the beginning of the episode. It will just make all of its coins basically by selling wood. And agent three will make all of its coins by selling stone. And agent two will collect both and sell both. And agent one will just spend money in trading. So you'll have a specialization here. Agent one, which is the highest skill one right here, will buy resources in order to build more houses because it clearly profits from building lots and lots and lots and lots of houses. So it will use that money to buy more resources rather than go and collecting them. While all the other ones basically forgo building houses in favor of they just collect the resources and they just trade them way to the agent one that's more profitable for them than building houses themselves. So you see this kind of specialization emerging in these games, which I find, I find this to be pretty cool that you see something like this, like a really stark division of labor emerging just from these very, very small set of rules. And you can analyze this game in different ways. They have a few more plots where this becomes quite apparent that sorry, that these agents specialize. So you see here resources collected, sorry about that, resources collected. If you have the lowest skill and the highest skill labors, the lowest skills, they mainly, this should be a 10. They mainly collect resources, while the highest skill labor mainly goes for building things. It doesn't collect resources, but net income from building is really high while everyone else just doesn't build at all. All right, so we have a division of labor emerging. Now this was a free market. Let's actually compare the different algorithms. So if you look at social welfare, this is this thing here, equality times productivity. You can see that the AI economist will outperform over time over the training progress, it will outperform all of the other systems. So it will outperform the free market, the US federal tax system, and the SAS formula if trained for long enough, which is to be expected, right? If you put RL onto a cost function, it will then optimize that cost function. But it's pretty cool to see that there's a lot of headroom here over what we currently have. Now let's look at some of the strategies it comes up with. So what do these games look like where the AI has imposed different tax strategies? So this is with the SAS strategy. You can see that here. Again, you see this inequality emerging with the yellow player here building most of the houses. With the AI economist, again, there is inequality, but you can see at the distribution that agent one only ends up with about half of the wealth, where if you compare this to the free market here, then agent one ends up with like two thirds of the wealth, right? This is the game we saw before. But there is not qualitatively that much of a difference, but there is in the end result. All right, let's look at what these policies actually come up with. So what is the tax policy that the AI comes up with? So this tax policy outperforms on this social welfare metric. And this is very interesting, right? So first of all, you see that it's right zigzag. It's like down, up, down, up, which is already weird. So the first very weird thing is the spike at the very bottom. So that thing here, what's that thing here? Those are the poorest people in your society, and you're taxing them the highest. Right? So just imagine this, you're here downtrodden by life, abandoned by society, you have no money, no house, no nothing. And you're just trying to get a job, you're just getting like a little bit of money. And you can buy a cheeseburger, and then the government comes. Give us that. Give us that money. Come on. So basically, these are the poor. And the poor in this system is just F you. F you the poor. Now, the reason why this happens is pretty clear, right? The reason why this happens is because you want to encourage people to go here to earn more money. So it's not like the government makes any money from the poor people independently of how high it taxes them. But it is basically an incentive structure to make them move over to the somewhat more productive population. Because here it's assumed kind of that even the lowest skilled ones can move over a bit if you just tax them enough at the low brackets, right? So this is what I find to be you just have to realize that it is so hard, I believe it is almost impossible to encapsulate what we really want in a system into a formula to be into a cost function to be optimized. It is so incredibly hard. And you see that here, of course, it is going to result in a better social outcome, but it just doesn't feel right to tax the poor at what 60%? Okay, so F the poor, right? And then you get to this level right here. And interestingly, if you earn even more, you'll be taxed high again, right? So this, we're kind of used to that. You earn little, you pay little, you earn more. You pay more. But then comes this entire valley here. What's up with that? Right? Like WT, and this can be this is now of course, the same reasoning as you have with this size formula here is where the rich people, you want to tax them less so that they are more productive such that they generate more coins. And even though you tax them less percentage wise, they will end up paying more money in absolute terms. Because because you basically encourage them to produce more. So that is that is kind of that is the, I guess the reasoning behind this. But what you have to wreck, you have to recognize what's happening here, right? What are we optimizing? We're optimizing this productivity times equality. And what do we get? You see, you get two big valleys of attraction, one here, and one here. And that means that this algorithm favors a two class society. Right? And I believe this is this is partially the limitations of this simulation here, the fact that you only have four agents, the fact that you can only do two things either collect or build, right? It encourages a two class society, this specialization that you saw, right? So you say these here are the moneymakers, right? And these here are the collectors. And it is very hard to move from one group to the other. Because if you you earn more coins as a collector, you're here, and you're really discouraged here. If you move there, you want to move all the way over here, right? Now, the people that are already over here, if they earn an extra coin, that doesn't bother them too much. So they're very encouraged to earn more money. But the very, the poorer people on this side, they're basically discouraged from earning more money, because the system needs them to stay at that collector level, right? So the system encourages the two class society because we have not built social mobility into the into the into the equation, we have not built a measure for social social mobility into the cost function. And therefore, the AI doesn't care that the poor people will stay poor and rich people will stay rich. It just knows that this is the best outcome for society overall, given the cost function that we had, again, this just doesn't seem like fair to us, like what we want, we want someone to be able to make it over here, right, even if they start out from the bottom. And so we'd have to we have to build that in. So we have a system that is effing eff the poor, right? No social mobility, mobility. No. And then what's happening at the end? What's happening at the end? This is beautiful. Very rich people. These are the moneymaker, right? This is the this is the monopoly guy top hat monocle wearing Scrooge McDuck bathing in coins. This is where the the government makes their money. And the discrepancy is really stunning, because you could also argue, hey, why don't we apply the same reasoning as we applied here and here? Why is not is it not like the case that if the rich people if you tax them lower, they'll pay more money and so on. I believe again, this might be just a result of this, how the simulation is set up. So we'll move away quickly and we'll come back to this. Here's what I find particularly interesting about this paper, which just confuses the heck out of me. It is a double periodic game. So it's an inner outer loop game. What do I mean by that? They have these episodes, right? Here is the start. And here is the end. And they subdivide this into, as we said, 1000 steps. So an agent is here and it can do step, step, step, step, step, and it can perform these actions. This is the agent. There are 1000 steps here and the agent just tries to collect as much coins. So this is your classic RL problem. But also they divide this into 10, what they call periods. And I'm just going to draw maybe four periods, right? So this thing here, they call one period where the whole thing is an episode. Now the purpose of the period is that at the beginning of each period, the government, the government can impose a new tax schedule. So the government doesn't only fix the taxes once, but it can change the taxes over the course of the episode, right? Now this is what I find. I just don't see why. So now you're formulating the tax giving objective as a sequential decision making. It's like the government saying, well, today we have high taxes, but tomorrow we have low taxes and the day after that we have high taxes again. And it just doesn't make sense for any government to do this. What you should do is you should set taxes once at the beginning of the episode and then see how that turns out and then try to maximize your tax schedule. Because all we're looking at, we're only ever looking at how the taxes are at the end, right? The things that we've examined are just the last taxes that the AI has issued. We don't know the dynamic of what happens in between. This might be super wild actually, what the AI does in between. And I just don't see the framing as a sequential decision problem. And I believe this is just an over engineered thing. Because someone wanted a reason and here is the architecture, right? You see someone wanted a reason to put an LSTM in there. Someone is thinking like, well, RL, that means like sequential decisions and so on. And RL in this outer loop, the way I propose it would just be a one step per episode decision, which is a bandit problem. And as we all know, bandits are boring. So they didn't want this to be a bandit problem. They wanted to be a sequential problem. And that's why they made this period thing, which I find dumb. So another factor here, and I'm going to tell you how this relates to the to the weird rich people are taxed high. Another factor here is look at this. It's a CNN, an MLP, an LSTM and an MLP and the agent as well. And I can tell you right now, the CNN has two layers. Two. And the LSTM has like 128 units in its hidden state. So these are tiny, tiny models. And it is not a model based RL, it's model free RLs, proximal policy optimization. And the the the ability of these agents or planner to learn anything substantial here, I believe is just not super duper well, right. So the I believe that these are rather dumb agents. And you can see the tax rates given by the planner is fed into the agent model. But I don't think that the agent given such a small model can actually adjust to these inputs because you have to do some pretty good logic in order to from these tax brackets to determine how you should act right now. What I think is happening is the agent just kind of is aware of its skill level and through its rewards, it's trying to maximize its future rewards. And then when the government changes the tax rate, it will not, I'm almost positive it will not directly change its response to that. But it will kind of observe that something's happening in the world and then adjust maybe a little bit its overall strategy, but not in that particular instance, and it will be delayed or it will be like an overall strategy. And this might be one of the reasons why the tax brackets here might be screwed up because who says who says if I were this AI, what I could do is in period one through nine, I make the taxes really low for the rich people. So I just encourage everyone to make more money, right? Like come on, become more productive and I get the benefits of that. And then in the last episode, last period, right, I just freaking jack up that final tax bracket. It's like you, you have lots of money, give it to me. And then you just redistribute what you got there to the poor people in the very last period and thereby you achieve your goal of this social welfare function. But of course, this is not sustainable because all the rich people would just be kind of screwed through that and move down again, but it's the end of the episode. So what are they going to do? So I think the fact how this is framed, that there are just two different ways to get coins. But the fact that this is this periodical nature of the outer loop all might lead to something that becomes slowly more and more and more uninterpretable. Still cool though. All right, so the final thing, they do this with humans. Yes, real humans. So they let humans try it and they have this interface here and the humans, they behave quite differently from the AI. So there are a few different things where the humans act. But look at that here, AI economists, this is what the agents do, right? So this AI economist is the tax strategy. They just take these developed tax strategies and let the humans be the agents so that the you just want to observe how the agents act and whether or not the tax strategies also work when it's real humans acting in this environment and not our agents. So compare this to how the humans act. The humans they just build their houses in like neat little packets or straight lines or stuff like this. I just find it to be very funny. Now there are some things lacking in the human environment which I find really important. So first of all, they have no cost for moving, which I guess is minor. But second of all, they have no trade. And I think that just kills the whole experiment. Because now of course what you're going to get is the wealth is just going to be proportional to how much you get coins per house, which is different for each agent, right? So to me that that is now a pointless experiment if you can't trade because the outcome is just predictable. And I don't think that the human behavior changes in response to the different tax brackets. I think they'll just do and however they can make money, they'll make money, they'll build more houses until it becomes unprofitable. And that's it. So I don't see the I don't see the value of these experiments, even though they show that again, the AI economist outperforms the other tax strategies in this equality times productivity metric and also in another metric that they measure. The second problem I have is for the human experiments, they take this distribution here, they say, well, the AI, this is one of the distributions that the AI came up with. But you notice the lack of the F you poor people, and the lack of this big spike here for the rich people, which I find are one of the two features of the other distribution. So I think there's quite a bit of variance in what this AI comes up with. Or maybe it's just because this is periodical. But this is really confusing because they show and discuss that other distribution. And now all of a sudden, they say, well, we use this distribution that was also created by our AI. And it seems to be qualitatively quite different. In any case, let's look at how the humans behave under the different strategies. So in the size formula, you'll see that the light blue person here is kind of spreading out a bit, probably playing correctly. Everyone else is just neatly building their houses. Humans are so territorial. And most of them, they kind of stay in their little corner. And they're like, this is my corner. I'm going to build my houses here in a nice thing. And under the AI economist, again, you don't really see a different thing just because the taxes are different. The qualitative behavior is quite the same. It's just building straight lines. And I think the difference is more between the humans. So I think it's not always the same humans. And the difference might be more between the humans. And you kind of see that humans clearly haven't really trained or discovered the optimal strategy. They're just doing something. And what you're seeing is just a result of the taxation. It's not different behavior. And this here, this is the best. Okay, watch the on the bottom right, the human. They're just first they do something, they're just walling up the other players. And look, this is this is the best. I am going to build a big beautiful wall. And I'm going to have the orange guy pay for it. It's Donald Trump in the game. Amazing. And look at the end, they actually managed to lock in the other players so they can't move anymore. Donald Trump wins. Amazing. And though, actually, the yellow player appears to win economy wise. But what do you want with lots of money if you can't move? So I again, I find these human experiments to be rather pointless here because you disable trade and you don't train the humans to find a good strategy. Alright, but in that, I find the entire paper to be pretty cool code is going to be released, they promise and they have checked that they have no ethical problems. Of course, I invite you to check out the paper. If you like content like this, please subscribe, share and leave a comment of what you think. Thank you so much for listening and bye bye.
 Alright, today we're going to find out why AI is much better at governing people, why
 poor people really should pay more taxes, and how Donald Trump is just a normal human.
 Alright, we'll dive into it.
 We're looking at the AI Economist by Salesforce Research.
 Now Salesforce Research has kind of created a simulated world environment where they can
 place agents in it and the agents, they can move around, they can collect resources, they
 can trade those resources, and they can use those resources to build houses and that will
 earn them coins.
 And each agent wants to maximize its own coins, but also there's the government and the government
 can set taxes.
 So they collect money from everyone and they redistribute it.
 And the goal now is going to be that the AI handles both the agent and the taxes and
 we want to maximize the social welfare of the entire population.
 Alright, that's the goal.
 So the paper here is called The AI Economist Improving Equality and Productivity with AI
 Driven Tax Policies by Stefan Cheng and Alexander Trott and other people from Salesforce Research
 and Harvard University.
 So as I said, this is a simulated environment and the simulated environment works like this.
 There is a 2D plane, kind of like a game playing field and in this game there are agents.
 Here you can see the agents, there are always four agents.
 Where?
 Oh, down here.
 What are you doing in the corner?
 Come on, be productive.
 The agents are in this world and they can do certain things.
 They have certain actions at their disposal.
 So first of all, they can move around.
 They can move down, left, right and so on.
 Whenever they walk past a resource tile, they collect the resource.
 This is stone and this is wood.
 So there are two kinds of resources.
 And then the last actions the agents have is building a house.
 One wood and one stone will create one house and the house gives you coins.
 So this is a house and that will give you coins.
 But how much coins you get is different from agent to agent and this represents the agents
 different skill levels.
 This is an abstraction and the kind of economic theory behind it is that the income inequality
 in people, one of the main drivers of it is that they are skilled differently and therefore
 are able to convert one unit of labor into more money than another lower skilled worker.
 So this is here represented by the fact that maybe if this agent here builds the house,
 they'll get 50 coins.
 But if this agent here would build the same house, they'll only get 10 coins.
 So we'll call this here a high skilled worker and this here a low skilled worker.
 Now the last thing, sorry, I saw the last thing before, but the very last thing the
 agents can do is they can trade.
 So if one agent has too many resources and the other one has not enough, they can trade
 those resources among each other for those coins.
 So once you build a house, you collect some coins, you can then either go and collect
 more resources or you can use those coins in order to buy resources off of other people.
 This is unlucky.
 No coins, no houses, and no resources.
 Look at them.
 Oh yeah, so you also can't move across the water here.
 You can only move on the grass.
 You can also not move through a house, which gives you some interesting abilities because
 you can just build a house right here.
 And yeah, so and you can't move over other players.
 But these are the rules are pretty simple.
 And the goal here is for the agents to maximize the number of coins they get in 1000 steps.
 So the number H here is 1000, which is the number of steps that the agents can take before
 the game is over and it restarts again.
 So each agent is using reinforcement learning in order to learn how to achieve the maximum
 number of coins.
 Now the policy is of course going to be different depending on whether that is a high or a low
 skilled worker.
 The catch here is that outside of this there is the government, the government here, let's
 draw this big house with the flag of our fictitious nation, which is like this.
 That's the flag.
 And the government will observe what's happening here and they will issue a tax taxes.
 So it will issue a tax distribution.
 Now how do you imagine that?
 So if you imagine the government says something like this for the first 10 coins you own,
 you owe us 5% of that.
 For the next 10 coins, so from 10 to 20 you earn, you owe us 10% and so on.
 So if you earn even more, you owe us more and more percent of those extra coins.
 This is what you might know as a progressive tax schedule.
 The more you earn, the more percentage wise you pay on that extra earned money.
 This is what you might be used to, but there are other tax schedules and the exact histogram
 you see or the exact how many percent for which amount of coins, that is the action
 of the government.
 So the government decides on the taxes and the taxes are just collected from the income.
 So if an agent earns these coins, then it has to pay taxes to the government and the
 government will redistribute all the taxes it has collected equally among the population.
 So if you pay a lot, you might lose through this process and if you just pay a little
 taxes you might gain through this process.
 So that's it.
 That is the basic premise of the game.
 The agents are using reinforcement learning and I believe the newness of this paper is
 also that the government now is using reinforcement learning in order to determine the optimal
 tax policy.
 There is kind of this inner loop here and there is this outer game where the government
 also tries to maximize the RL.
 And what does the government try to maximize?
 Good question.
 It is a measure that's called social welfare.
 Now social welfare consists of two things and they have this here way down in the paper.
 Social welfare in this paper consists of two things.
 First of all, economic productivity, which basically just means how many coins has anyone
 produced.
 It doesn't matter who, but just the total amount of coins produced.
 The second one is income equality and this is related to the Gini index.
 So if you plot the cumulative distribution of wealth, a fully equal society would be
 a straight line because 50% of the people would have 50% of the money and so on.
 But almost all true societies have something like this where 50% of the people might have
 10% of the money and the rest 50% of the people has the other 90%.
 And the measure of inequality is this area here.
 This is called the Gini index and 1 minus this area is what this paper has as an equality
 measure.
 So the higher this number, the more equal is the society in terms of their income distribution.
 Now what is actually optimized for here is this thing, equality times productivity.
 So you want both to be high, your income equality and your productivity.
 There's a trade off here of course, but you can have multiple ways to trade that off and
 that will give you the different thing.
 They call this the social welfare function.
 And that's the thing that the government RL agent optimizes for.
 So you can see here already the free market, even though it's the most productive, produces
 the most coins because if you have a free market means no taxes.
 If you have no taxes, then people are basically encouraged to earn more money because they
 don't have to pay taxes on them.
 As soon as you tax them, they're less encouraged to earn more money.
 And therefore if you have no taxes, the most coins will be earned in total.
 But the equality suffers.
 So the equality is the lowest among these things considered.
 If you compare that to the AI economist, the AI economist achieves the highest social welfare.
 It achieves the highest equality, but it doesn't suffer as much in productivity as other systems
 here.
 And the baseline systems are first of all, the US federal system.
 This is not particularly tied to the US.
 This is basically every system or most of the systems that you have currently in the
 world is the progressive tax system and the SAES formula, which I believe is an economically
 theory based system, which is a regressive tax schedule.
 You can see them down here where the US federal will be progressive, means the more you earn,
 the more percentage wise you pay.
 While the SAES formula will be regressive, which generally means the more you earn, the
 less you pay.
 I believe this was derived under some assumptions to be the optimal tax distribution.
 And the AI economist will come to this in a second.
 Let's actually just look at one of these things first, one of these games, how this plays
 out.
 The cool thing here is that they have pretty flashy animations.
 So you can look how does one of these games turn out.
 Now this is a free market game and you can see the agents moving around collecting things,
 building houses.
 And you might notice that one of the agents, namely agent one, is just building all of
 the houses.
 And generally just kind of being a dick, being in everyone's face and kind of building things
 everywhere.
 And the other ones don't.
 Or just very few, like the light blue on the bottom left builds some houses.
 On the right you can see how the distribution of wealth is structured.
 And you see agent one ends up with most of the wealth.
 Now the size of the circle I think is the total productivity.
 So you can see this grows over time mainly because agent one becomes so rich.
 And if you analyze this, if you analyze what's happening here, then you'll see that agent
 one and I might be...
 Yeah, they have a graph up here.
 So it is very interesting what happens.
 This is kind of the same game.
 So agent one here is this orange dot and agents two, three and four are these dots here.
 And this graph here is coin from trading.
 So how much money they win or lose from trading.
 Now the green bars are trading wood and the brown bars are trading stone.
 So you see agent number four, which is the lowest skilled, the skill is just determined
 at the beginning of the episode.
 It will just make all of its coins basically by selling wood.
 And agent three will make all of its coins by selling stone.
 And agent two will collect both and sell both.
 And agent one will just spend money in trading.
 So you'll have a specialization here.
 Agent one, which is the highest skill one right here, will buy resources in order to
 build more houses because it clearly profits from building lots and lots and lots and lots
 of houses.
 So it will use that money to buy more resources rather than go and collecting them.
 While all the other ones basically forgo building houses in favor of they just collect the resources
 and they just trade them way to the agent one that's more profitable for them than building
 houses themselves.
 So you see this kind of specialization emerging in these games, which I find, I find this
 to be pretty cool that you see something like this, like a really stark division of labor
 emerging just from these very, very small set of rules.
 And you can analyze this game in different ways.
 They have a few more plots where this becomes quite apparent that sorry, that these agents
 specialize.
 So you see here resources collected, sorry about that, resources collected.
 If you have the lowest skill and the highest skill labors, the lowest skills, they mainly,
 this should be a 10.
 They mainly collect resources, while the highest skill labor mainly goes for building things.
 It doesn't collect resources, but net income from building is really high while everyone
 else just doesn't build at all.
 All right, so we have a division of labor emerging.
 Now this was a free market.
 Let's actually compare the different algorithms.
 So if you look at social welfare, this is this thing here, equality times productivity.
 You can see that the AI economist will outperform over time over the training progress, it will
 outperform all of the other systems.
 So it will outperform the free market, the US federal tax system, and the SAS formula
 if trained for long enough, which is to be expected, right?
 If you put RL onto a cost function, it will then optimize that cost function.
 But it's pretty cool to see that there's a lot of headroom here over what we currently
 have.
 Now let's look at some of the strategies it comes up with.
 So what do these games look like where the AI has imposed different tax strategies?
 So this is with the SAS strategy.
 You can see that here.
 Again, you see this inequality emerging with the yellow player here building most of the
 houses.
 With the AI economist, again, there is inequality, but you can see at the distribution that agent
 one only ends up with about half of the wealth, where if you compare this to the free market
 here, then agent one ends up with like two thirds of the wealth, right?
 This is the game we saw before.
 But there is not qualitatively that much of a difference, but there is in the end result.
 All right, let's look at what these policies actually come up with.
 So what is the tax policy that the AI comes up with?
 So this tax policy outperforms on this social welfare metric.
 And this is very interesting, right?
 So first of all, you see that it's right zigzag.
 It's like down, up, down, up, which is already weird.
 So the first very weird thing is the spike at the very bottom.
 So that thing here, what's that thing here?
 Those are the poorest people in your society, and you're taxing them the highest.
 Right?
 So just imagine this, you're here downtrodden by life, abandoned by society, you have no
 money, no house, no nothing.
 And you're just trying to get a job, you're just getting like a little bit of money.
 And you can buy a cheeseburger, and then the government comes.
 Give us that.
 Give us that money.
 Come on.
 So basically, these are the poor.
 And the poor in this system is just F you.
 F you the poor.
 Now, the reason why this happens is pretty clear, right?
 The reason why this happens is because you want to encourage people to go here to earn
 more money.
 So it's not like the government makes any money from the poor people independently of
 how high it taxes them.
 But it is basically an incentive structure to make them move over to the somewhat more
 productive population.
 Because here it's assumed kind of that even the lowest skilled ones can move over a bit
 if you just tax them enough at the low brackets, right?
 So this is what I find to be you just have to realize that it is so hard, I believe it
 is almost impossible to encapsulate what we really want in a system into a formula to
 be into a cost function to be optimized.
 It is so incredibly hard.
 And you see that here, of course, it is going to result in a better social outcome, but
 it just doesn't feel right to tax the poor at what 60%?
 Okay, so F the poor, right?
 And then you get to this level right here.
 And interestingly, if you earn even more, you'll be taxed high again, right?
 So this, we're kind of used to that.
 You earn little, you pay little, you earn more.
 You pay more.
 But then comes this entire valley here.
 What's up with that?
 Right?
 Like WT, and this can be this is now of course, the same reasoning as you have with this size
 formula here is where the rich people, you want to tax them less so that they are more
 productive such that they generate more coins.
 And even though you tax them less percentage wise, they will end up paying more money in
 absolute terms.
 Because because you basically encourage them to produce more.
 So that is that is kind of that is the, I guess the reasoning behind this.
 But what you have to wreck, you have to recognize what's happening here, right?
 What are we optimizing?
 We're optimizing this productivity times equality.
 And what do we get?
 You see, you get two big valleys of attraction, one here, and one here.
 And that means that this algorithm favors a two class society.
 Right?
 And I believe this is this is partially the limitations of this simulation here, the fact
 that you only have four agents, the fact that you can only do two things either collect
 or build, right?
 It encourages a two class society, this specialization that you saw, right?
 So you say these here are the moneymakers, right?
 And these here are the collectors.
 And it is very hard to move from one group to the other.
 Because if you you earn more coins as a collector, you're here, and you're really discouraged
 here.
 If you move there, you want to move all the way over here, right?
 Now, the people that are already over here, if they earn an extra coin, that doesn't bother
 them too much.
 So they're very encouraged to earn more money.
 But the very, the poorer people on this side, they're basically discouraged from earning
 more money, because the system needs them to stay at that collector level, right?
 So the system encourages the two class society because we have not built social mobility
 into the into the into the equation, we have not built a measure for social social mobility
 into the cost function.
 And therefore, the AI doesn't care that the poor people will stay poor and rich people
 will stay rich.
 It just knows that this is the best outcome for society overall, given the cost function
 that we had, again, this just doesn't seem like fair to us, like what we want, we want
 someone to be able to make it over here, right, even if they start out from the bottom.
 And so we'd have to we have to build that in.
 So we have a system that is effing eff the poor, right?
 No social mobility, mobility.
 No.
 And then what's happening at the end?
 What's happening at the end?
 This is beautiful.
 Very rich people.
 These are the moneymaker, right?
 This is the this is the monopoly guy top hat monocle wearing Scrooge McDuck bathing in
 coins.
 This is where the the government makes their money.
 And the discrepancy is really stunning, because you could also argue, hey, why don't we apply
 the same reasoning as we applied here and here?
 Why is not is it not like the case that if the rich people if you tax them lower, they'll
 pay more money and so on.
 I believe again, this might be just a result of this, how the simulation is set up.
 So we'll move away quickly and we'll come back to this.
 Here's what I find particularly interesting about this paper, which just confuses the
 heck out of me.
 It is a double periodic game.
 So it's an inner outer loop game.
 What do I mean by that?
 They have these episodes, right?
 Here is the start.
 And here is the end.
 And they subdivide this into, as we said, 1000 steps.
 So an agent is here and it can do step, step, step, step, step, and it can perform these
 actions.
 This is the agent.
 There are 1000 steps here and the agent just tries to collect as much coins.
 So this is your classic RL problem.
 But also they divide this into 10, what they call periods.
 And I'm just going to draw maybe four periods, right?
 So this thing here, they call one period where the whole thing is an episode.
 Now the purpose of the period is that at the beginning of each period, the government,
 the government can impose a new tax schedule.
 So the government doesn't only fix the taxes once, but it can change the taxes over the
 course of the episode, right?
 Now this is what I find.
 I just don't see why.
 So now you're formulating the tax giving objective as a sequential decision making.
 It's like the government saying, well, today we have high taxes, but tomorrow we have low
 taxes and the day after that we have high taxes again.
 And it just doesn't make sense for any government to do this.
 What you should do is you should set taxes once at the beginning of the episode and then
 see how that turns out and then try to maximize your tax schedule.
 Because all we're looking at, we're only ever looking at how the taxes are at the end, right?
 The things that we've examined are just the last taxes that the AI has issued.
 We don't know the dynamic of what happens in between.
 This might be super wild actually, what the AI does in between.
 And I just don't see the framing as a sequential decision problem.
 And I believe this is just an over engineered thing.
 Because someone wanted a reason and here is the architecture, right?
 You see someone wanted a reason to put an LSTM in there.
 Someone is thinking like, well, RL, that means like sequential decisions and so on.
 And RL in this outer loop, the way I propose it would just be a one step per episode decision,
 which is a bandit problem.
 And as we all know, bandits are boring.
 So they didn't want this to be a bandit problem.
 They wanted to be a sequential problem.
 And that's why they made this period thing, which I find dumb.
 So another factor here, and I'm going to tell you how this relates to the to the weird rich
 people are taxed high.
 Another factor here is look at this.
 It's a CNN, an MLP, an LSTM and an MLP and the agent as well.
 And I can tell you right now, the CNN has two layers.
 Two.
 And the LSTM has like 128 units in its hidden state.
 So these are tiny, tiny models.
 And it is not a model based RL, it's model free RLs, proximal policy optimization.
 And the the the ability of these agents or planner to learn anything substantial here,
 I believe is just not super duper well, right.
 So the I believe that these are rather dumb agents.
 And you can see the tax rates given by the planner is fed into the agent model.
 But I don't think that the agent given such a small model can actually adjust to these
 inputs because you have to do some pretty good logic in order to from these tax brackets
 to determine how you should act right now.
 What I think is happening is the agent just kind of is aware of its skill level and through
 its rewards, it's trying to maximize its future rewards.
 And then when the government changes the tax rate, it will not, I'm almost positive it
 will not directly change its response to that.
 But it will kind of observe that something's happening in the world and then adjust maybe
 a little bit its overall strategy, but not in that particular instance, and it will be
 delayed or it will be like an overall strategy.
 And this might be one of the reasons why the tax brackets here might be screwed up because
 who says who says if I were this AI, what I could do is in period one through nine,
 I make the taxes really low for the rich people.
 So I just encourage everyone to make more money, right?
 Like come on, become more productive and I get the benefits of that.
 And then in the last episode, last period, right, I just freaking jack up that final
 tax bracket.
 It's like you, you have lots of money, give it to me.
 And then you just redistribute what you got there to the poor people in the very last
 period and thereby you achieve your goal of this social welfare function.
 But of course, this is not sustainable because all the rich people would just be kind of
 screwed through that and move down again, but it's the end of the episode.
 So what are they going to do?
 So I think the fact how this is framed, that there are just two different ways to get coins.
 But the fact that this is this periodical nature of the outer loop all might lead to
 something that becomes slowly more and more and more uninterpretable.
 Still cool though.
 All right, so the final thing, they do this with humans.
 Yes, real humans.
 So they let humans try it and they have this interface here and the humans, they behave
 quite differently from the AI.
 So there are a few different things where the humans act.
 But look at that here, AI economists, this is what the agents do, right?
 So this AI economist is the tax strategy.
 They just take these developed tax strategies and let the humans be the agents so that the
 you just want to observe how the agents act and whether or not the tax strategies also
 work when it's real humans acting in this environment and not our agents.
 So compare this to how the humans act.
 The humans they just build their houses in like neat little packets or straight lines
 or stuff like this.
 I just find it to be very funny.
 Now there are some things lacking in the human environment which I find really important.
 So first of all, they have no cost for moving, which I guess is minor.
 But second of all, they have no trade.
 And I think that just kills the whole experiment.
 Because now of course what you're going to get is the wealth is just going to be proportional
 to how much you get coins per house, which is different for each agent, right?
 So to me that that is now a pointless experiment if you can't trade because the outcome is
 just predictable.
 And I don't think that the human behavior changes in response to the different tax brackets.
 I think they'll just do and however they can make money, they'll make money, they'll build
 more houses until it becomes unprofitable.
 And that's it.
 So I don't see the I don't see the value of these experiments, even though they show that
 again, the AI economist outperforms the other tax strategies in this equality times productivity
 metric and also in another metric that they measure.
 The second problem I have is for the human experiments, they take this distribution here,
 they say, well, the AI, this is one of the distributions that the AI came up with.
 But you notice the lack of the F you poor people, and the lack of this big spike here
 for the rich people, which I find are one of the two features of the other distribution.
 So I think there's quite a bit of variance in what this AI comes up with.
 Or maybe it's just because this is periodical.
 But this is really confusing because they show and discuss that other distribution.
 And now all of a sudden, they say, well, we use this distribution that was also created
 by our AI.
 And it seems to be qualitatively quite different.
 In any case, let's look at how the humans behave under the different strategies.
 So in the size formula, you'll see that the light blue person here is kind of spreading
 out a bit, probably playing correctly.
 Everyone else is just neatly building their houses.
 Humans are so territorial.
 And most of them, they kind of stay in their little corner.
 And they're like, this is my corner.
 I'm going to build my houses here in a nice thing.
 And under the AI economist, again, you don't really see a different thing just because
 the taxes are different.
 The qualitative behavior is quite the same.
 It's just building straight lines.
 And I think the difference is more between the humans.
 So I think it's not always the same humans.
 And the difference might be more between the humans.
 And you kind of see that humans clearly haven't really trained or discovered the optimal strategy.
 They're just doing something.
 And what you're seeing is just a result of the taxation.
 It's not different behavior.
 And this here, this is the best.
 Okay, watch the on the bottom right, the human.
 They're just first they do something, they're just walling up the other players.
 And look, this is this is the best.
 I am going to build a big beautiful wall.
 And I'm going to have the orange guy pay for it.
 It's Donald Trump in the game.
 Amazing.
 And look at the end, they actually managed to lock in the other players so they can't
 move anymore.
 Donald Trump wins.
 Amazing.
 And though, actually, the yellow player appears to win economy wise.
 But what do you want with lots of money if you can't move?
 So I again, I find these human experiments to be rather pointless here because you disable
 trade and you don't train the humans to find a good strategy.
 Alright, but in that, I find the entire paper to be pretty cool code is going to be released,
 they promise and they have checked that they have no ethical problems.
 Of course, I invite you to check out the paper.
 If you like content like this, please subscribe, share and leave a comment of what you think.
 Thank you so much for listening and bye bye.