Reward Is Enough (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deep learning tutorial introduction to deep learning what is deep learning how to achieve agi artificial general intelligence how to create intelligence reward maximisation reward maximization reinforcement learning is alphago intelligence is gpt 3 self aware is gpt 3 intelligent how to create ai how to achieve ai general ai agent environment deepmind
#reinforcementlearning #deepmind #agi

What's the most promising path to creating Artificial General Intelligence (AGI)? This paper makes the bold claim that a learning agent maximizing its reward in a sufficiently complex environment will necessarily develop intelligence as a by-product, and that Reward Maximization is the best way to move the creation of AGI forward. The paper is a mix of philosophy, engineering, and futurism, and raises many points of discussion.

OUTLINE:
0:00 - Intro & Outline
4:10 - Reward Maximization
10:10 - The Reward-is-Enough Hypothesis
13:15 - Abilities associated with intelligence
16:40 - My Criticism
26:15 - Reward Maximization through Reinforcement Learning
31:30 - Discussion, Conclusion & My Comments

Paper: https://www.sciencedirect.com/science/article/pii/S0004370221000862

Abstract:
In this article we hypothesise that intelligence, and its associated abilities, can be understood as subserving the maximisation of reward. Accordingly, reward is enough to drive behaviour that exhibits abilities studied in natural and artificial intelligence, including knowledge, learning, perception, social intelligence, language, generalisation and imitation. This is in contrast to the view that specialised problem formulations are needed for each ability, based on other signals or objectives. Furthermore, we suggest that agents that learn through trial and error experience to maximise reward could learn behaviour that exhibits most if not all of these abilities, and therefore that powerful reinforcement learning agents could constitute a solution to artificial general intelligence.

Authors: David Silver, Satinder Singh, Doina Precup, Richard S. Sutton

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 From the makers of Is All You Need and Do We Really Need and Is It Even Useful now comes Enough. So, today we're going to look at Reward Is Enough by David Silver, Satinder Singh, Doina Prakup and Richard S. Sutton. This paper is a more philosophical paper, I feel, though it presents itself as having practical advice in it. And the core hypothesis in this paper, and they state it as a hypothesis, is that maximizing reward in a sufficiently complex environment is a sufficient condition for intelligence to arise implicitly in service of maximizing that reward. So the example they give is like a squirrel who wants to get as many nuts as possible, has to learn to do all kinds of things in the environment. In order to do that, it needs to know how to perceive, how to motor act in the world, it needs to understand maybe the cycles of the year, it needs to be able to communicate and fend away other squirrels and so on. So a lot of these abilities naturally arise from something that just wants to maximize a reward in a complex environment. I do have my troubles with this hypothesis right here, especially how they present it, but we'll go through the paper, look at the hypothesis, at the reasoning, and as always, tell me what you think about this work. The conclusion of the work is that if this is correct, this sort of gives a straight path to general intelligence, namely, let's just maximize reward in a sufficiently complex environment. And as always, if you do like it, share it out, subscribe if you haven't, and we'll dive into the paper. So the abstract says, in this article, we hypothesize that intelligence and its associated abilities can be understood as subserving the maximization of reward. Accordingly, reward is enough to drive behavior that exhibits abilities studied in natural and artificial intelligence, including knowledge, learning, perception, social intelligence, language, generalization, and imitation. This is in contrast to the view that specialized problem formulations are needed for each ability based on other signals or objectives. Furthermore, we suggest that agents learn through trial and error experience to maximize reward could learn behavior that exhibits most if not all of these abilities. So it's agents that learn through trial and error. And therefore, that powerful reinforcement learning agents could constitute a solution to artificial general intelligence. Now this has sort of this is kind of the deep mind ethos, right in a nutshell, it is let's just build in not like the most powerful reward maximization agents specifically through reinforcement learning that we can, and that will sort of get us to general intelligence because in order to achieve anything in the world, you need to be intelligent if you want to achieve it to a very, very high degree. Now if that tickles you a bit in the wrong spot, so it does the same to me. But so they contrast this here. They ask how does intelligent intelligence arise? How does it arise? And how is it so bountiful and so varied and has very different subsystems? And how does this come about? They say one possible answer is that each ability arises from the pursuit of a goal that is designed specifically to elicit that ability. So for example, the ability of social intelligence has often been framed as the Nash equilibrium of a multi agent system. And they go through others. In this paper, they say we consider an alternative hypothesis that the generic objective of maximizing reward is enough to drive behavior that exhibits most if not all abilities that are studied in natural and artificial intelligence. So they give an example right here with the with the squirrel. And so one example is a squirrel in sort of the natural world. And the other example is a kitchen robot or a household robot, also in the natural world. Now the one of the core points of this paper is that the environment needs to be let's say, complex enough. And I feel like they're only going to be satisfied with a particular environment than that is the real world. So if they say a complex environment, just think of the real world, like be that, you know, agents on the real internet in the real world, or be that squirrels in the actual physical world, they think of environments that are sufficiently complex. And that's sort of how this hypothesis draws their power. So the description of this figure says, the reward is enough hypothesis postulates that intelligence yada yada yada. For example, a squirrel acts as to maximize its consumption of food, that's the at the top right here, which is the reward depicted by the acorn the acorn symbol, or a kitchen robot acts as to maximize cleanliness. To achieve these goals, complex behaviors are required that exhibit a wide variety of abilities associated with intelligence. Okay, so the squirrel must learn to perceive it must learn to climb, it must learn to assess the knots, it must learn to bury them, it must learn to remember where they are, and so on. And the cleanliness robot must learn also to perceive to use its sort of movements, it must learn to wash. And it might even decide, let's get pizza delivered instead of instead of cooking, because that will be just cleaner, arguable. But yeah, so in in this framework, you can see on the right here, they see all of these different abilities, such as memory, perception, planning, and so on, just arising from these things, because they say, well, in order for the squirrel to maximize nuts, it needs to be able to do all of these things, otherwise, the squirrel will just sort of die. It can't, it can't, like without perceiving the nuts, it can't go get the nuts. And the also the cleanliness robot, if it is actually good at maximizing its reward, it needs to develop all these abilities, including right, like the social abilities in order to get a pizza delivered or in order to work together with the human, maybe even to manipulate the human to make less dirt. So that's the that's essentially the hypothesis right here. They do give some example. So they I mean, this first part, the introduction, I mean, you can read it for yourself, but they they say, they give these examples here, they say, watching this through the lens of reward maximization may, in fact, provide a deeper understanding since it explains why such ability arises, for example, avoidance of crocodiles, because you need you don't want to be eaten. In contrast, when each ability is understood as the solution to its own specialized goals, the why question is sidestepped in order to focus upon the what the ability does. Singular goal may provide a broader understanding. And it might even lead to new sort of new forms of intelligence. They give examples, of course, here, the games of go and chess, where just maximizing the reward alpha zero was able to come up with very new, very new tactics, very new openings and games and so on. And we didn't teach it to do openings, we didn't teach it to do board control and whatnot, or whatever they call in the things in go, we just asked it to maximize reward. And it came up with all of these sort of sub abilities by itself, right? Now they formalize this here, the reinforcement learning problem, they formalize it as an agent interacting with the environment. So here, the agent is just the decision making process. So in the squirrel, actually, only the squirrel brain would be the agent and the squirrel body is already part of the environment. So if you're in a sort of multi agent system, all the other agents are part of the environment in this framework. And the environment, you interact with it, and you get a reward signal, right? Reward signal, and then maximizing that reward signal, that is what you call reward maximization. And the core hypothesis of this paper, as I already said, right here, is the reward is enough hypothesis. And the hypothesis itself says, intelligence and its associated abilities can be understood as subserving the maximization of reward by an agent acting in its environment. It's a bit better stated above, I think that they say that the main different forms of intelligence can be understood as subserving the maximization of reward and that the many abilities associated with each each form of intelligence may arise implicitly from the pursuit of those rewards taken to its limit, we hypothesize that all intelligence and associated abilities may be understood in this manner. Now they do strengthen it. They do strengthen this hypothesis, because what you might be thinking of what I was thinking of first is that, oh, you know, you can just formulate any goal as reward. And that's what they say here, they say the reward hypothesis, which is different from their hypothesis, speculates that all goals of interest in studying natural or building artificial agents may be represented by rewards. This should not be confused with our reward is enough hypothesis, which considers the abilities that arise from the pursuit of any such any one such goal. Okay, so it's different than just saying, well, you can learn to perceive by doing reinforcement learning or well, you can learn to acquire knowledge by reinforcement learning. Now this is stronger. This says that the hypothesis here is intended to be much stronger, that intelligence and associated abilities will implicitly arise in the service of maximizing one of many possible reward signals corresponding to the many pragmatic goals towards which natural or artificial intelligence may be directed. So their idea is that there is a world, and that world is sort of complex enough, right? Maybe there's a tree, and you know, there's a house, so there is humans in it. And you have your little squirrel, whatever here, squirrel has a bushy tail and a head. I don't I don't know how squirrel looks just this is a head. And given in this environment, you pick any reward you can think of like any any reward signal, and then maximize such as like how many how much hunger do you have, you get that as a negative reward, and then maximizing that reward will lead implicitly to the squirrel having to develop intelligence having to develop perception having to develop the acquisition of knowledge, and even interacting with other squirrels or the humans in this world. This is a strong hypothesis. And as I said, I do have my problems with it. First though, they go through a bunch of things they say, well, let's explore how we let's explore some abilities that people naturally associate with intelligence. And let's explore how they might arise implicitly from reward maximization. Okay, so again, think of the squirrel wanting to get as many nuts as possible, or like, I don't know, a human wanting to survive and live and thrive in the real world, how something like intelligence may arise just as a product of maximizing that reward. And so here they go over a bunch of them. The first one is knowledge and learning. And the the arguments made here are always they're always pretty simple. They're they're giving you an example and saying, well, in order to maximize your reward in the real world, it's useful to have knowledge. And also because you don't have infinite memory or whatnot, it's useful to learn things and to abstract things right to to gather knowledge and so on. And then when here when they go for perception, they say, well, in order to maximize your reward to thrive, you need to perceive. Okay, so, you know, naturally, it's like almost a tautology. Okay, so they say, well, a reward maximization agent can reward maximize better if it perceives rather than if it doesn't perceive. Okay, so it's, it's, it's sort of and social intelligence. Yes. So if you're a human, you want to thrive in the world, it's better if you are socially intelligent. In fact, it's better if you know language because you can maximize reward by communicating. So language, if if you know might just be a byproduct of reward maximization, generalization. Well, it's better if you generalize and imitation. Yes, it's better if you imitate general intelligence. Well, if you want to reward maximize, you need to be able to instant sort of switch around between different sub goals in order to reward maximize and sort of solve new problems really easily. That would be really good in order for you to maximize your reward. And therefore general intelligence is might be, you know, if an if an agent maximized its reward, general intelligence will help. And I hope you've seen a little bit the trend here through all of these things. And I think especially in the last thing, in this general intelligence, the the flaw here, what I think is the flaw becomes rather obvious, because I mean, so reward is enough for for general intelligence. Essentially, you're saying, well, if we build something that's intelligent, right, then we have then intelligence is a byproduct of that. So if if you if you postulate your reward maximization as being intelligent, then yes, intelligence arises as a byproduct. Their their whole notion here is that if you have this complex environment, and you want to do anything, you need to be intelligent. And that's how they see the environment itself. The big question here is, of course, what is this environment? And what is the reward? And they have a discussion at the end where they say, well, as long as the environment is complex enough, we don't actually care, right? If it's complex enough, you know, the any and also for the reward, like any reward signal, any goal will do you can and they say, well, what if you if you're if your goal is to collect pebbles in the real world? Okay, so, you know, there is a pebble. There is a pebble. There is a pebble. So one agent might just learn to collect pebbles. But the other agent might learn to sort of use the internet and buy pebble collectors off of Amazon and then launch a political campaign and influence all the humans to also collect pebbles for itself and then influence everything and get rich and buy more pebbles. And that would necessitate intelligence. So just maximizing getting pebbles would sort of lead to intelligence. And I'm, I follow this way. But you know, again, this is sort of saying, if you're intelligent, then you're intelligent. And on the other hand, what if a agent could simply chemically transform anything it finds into pebbles or anything that's even possible? There's this this meme, right with the distribution, where here is the new guy. So here you have like, here we have this guy with this hair and with the teeth and this goes collect collect pebbles. And then here you have the I don't know, here's the smart person usually. And this person is like, well, influence all the people and buy things with money and do this and do that and do this and do that. And over here, I just imagine the the Zen. So there's usually the the person in the hoodie, right? The Zen person. Well, that's a terrible hoodie. The Zen person again going collect pebbles. Like you don't know this. I think this is such a this is such it's just kind of looking out at the world and then abstracting that into what they consider a reward of the environment. And then naturally tautologically, what will arise is that if you sort of maximize that, then intelligence will arise. And that's not even the end of it, right? Because a lot of things such as survival in the world and thriving in different environments are done without intelligence. If you think of bacteria, for example, bacteria, so I don't know. So here's the world. And there's like a tiny sliver where humans can live in about one fourth or so of that sliver. Yet bacteria, they're everywhere. Okay, they thrive much more than humans. So if the if the goal is survival and fitness, I mean, bacteria solve that problem completely without any intelligence. So I disagree that just reward maximization is enough. But then these people would say, well, the environment is not the same. The environment for a bacteria is not the same as for a human. Like if you are a human, clearly, your approach cannot be to just replicate. So if you're a bacteria, you know, here is here your bacteria, what do you do? You simply split. Cool. Don't need intelligence can colonize the entire planet. However, if you're a human, that is not an option. If you're a human, you need to be intelligent, right? Your environment is different. So your environment is much more what they would say complex, though I disagree, I think that bacteria's environment is incredibly complex. But the human environment, they would say is so complex. You as a human need intelligence in order to thrive that environment. Now again, there is a fallacy here, in my opinion, right in my opinion, what do I know? This is rich Sutton. But in my opinion, there is a fallacy here, namely, so there is the environment, right? And you're you're the human right here, you're in the environment. And in order to maximize your reward as a human, because you can't split because there are other humans around, you need intelligence, right? Intelligence needs to be right here in the human in order to survive and thrive in the human environment. However, that environment only exists because there is already intelligence, right? So first of all, you as a human, you don't acquire intelligence because you need it in your environment, you have it built into you. You do a bit of fine tuning during your life, but not like the no one doubts that a that intelligence is present even in a baby, okay, like it might not be able to, to act it out. But the all of the ingredients like the learning, the the ability to absorb knowledge and so on that like the ability to perceive and to to learn language that is all present already. So I disagree that humans acquire and have to acquire intelligence in order to thrive. Now they people would say, well, evolution, the evolutionary pressure on humans required intelligence and that might be true. But the individual human only needs intelligence because intelligence is already present in the environment, or if you want to call it differently. So here is your world and you can go into different niches, right? And one of the niches is the bacteria niche where you simply you simply split. Okay, another niche, another environmental niche is the niche where in fact you need intelligence in order to survive. But that is determined. That is just this niche, right? And you need intelligence because the other humans have intelligence. And because you were you're only born as a human, because you're because the the environment has or the evolutionary direction has pushed you into that direction. So it is not that the maximization of any reward be that fitness has led to intelligence because the maximization of that same reward has also not led to intelligence. It's simply that intelligence is present in this particular niche of the evolutionary process. Right? I see this as a clear distinction. Like I feel humans first of all, they have innate intelligence. And second of all, the environment is only such that intelligence is necessary because other humans before you also had intelligence. Nowhere in this process is the environment determinist or the driver of the development of intelligence. Because at the beginning, right here, the environment wasn't such that intelligence was necessary. So the environments and the intelligence they evolve together, sorry, the the environment that requires intelligence and the intelligent beings evolve together. At no point did you have an environment that required intelligence because of maximization of reward. And you had an object in that environment, not having intelligence and then having to acquire it. It's simply one niche. And there are other niches that don't require it. So that's, that's, that's my one of the largest things that I criticize right here, I disagree that reward maximization is enough for intelligence, because clearly the same reward maximization wasn't enough in other cases. Also, I think that there is no such like if they think of the real world, and agents with intelligence in it, those agents only exist because intelligence exists, not the other way around. The agents don't make intelligence, they already are intelligent for the most part. And the last thing right here is, I just want to point to you here that reward is enough for knowledge and learning. So now, they call learning one of these abilities that is associated with intelligence. And now we go to the next part. And the next part is where they ask themselves, well, given that we postulate that maximizing reward might be enough for intelligence, how should we achieve that? So it the hypothesis of maximization of reward is fully agnostic to the nature of the agent itself. This leaves open the important question on how to construct an agent that maximizes reward. So that's the question, right? How do you construct an agent that maximizes reward? Until now, we've heard no, of course, the answer is going to be reinforcement learning. But until now, we have actually not heard much of that except in examples. So they still leave it open how you would achieve such an agent. But now they're going to say reinforcement learning. But first they say, in this section, we suggest that this question may also be largely answered by reward maximization. Now I don't actually know whether this is intended here. But how to construct an agent that maximizes reward is largely answered by reward maximization. Like is this intended? Is this an intended back reference saying like, how do we construct x? Well x, like, is this, I'm not sure. Is this an intended like a little bit of a slight of like a little bit of a joke or something? I'm not sure. I'm not sure. I might just be too dumb, right? Specifically, we consider agents with the general ability to learn how to maximize the reward from their ongoing experience of interacting with the environment. Such agents we will refer to as reinforcement learning agents provide several advantages. So here they go into, you know, if you don't want to pre program, like you don't want to have the designer's knowledge of the environment be in there because the designer doesn't know everything, you want to actually let the agents learn themselves. And if the environment is sufficiently complex, and the reinforcement learning agent is sufficiently powerful, then it will like the richness of experience of a complex environment will provide enough signal for the agent, you know, disregard its practical implementation and sample complexity. Technically, the whole the whole richness of experience will provide enough of a signal to learn all of this. But I don't know, did you? There's another thing right here. We consider agents with a general ability to learn how to maximize reward. So how do we build reward maximization agents, which if successful will give rise to intelligence? Right? Well, by learning, okay. However, learning up here, learning is a product of intelligence or an ability that comes with intelligence, right? So like we need, we need learning in like learning comes with Intel learning is one of the abilities that indicates intelligence. So a little bit, it's like learning gens. So intelligence, if something is intelligent, right, it then then it will learn but also in order to achieve this intelligence through reward maximization, that's how we achieve intelligence but then in order to do reward maximization, we need a learning algorithm. But if the learning algorithm is not yet intelligent, right, then how is this happening? So I think you can I guess you can make a split and saying, well, this learning that we we use for reward maximization, that's sort of a learning that we design or something like this. But even if we design it, intelligence gives like if we design the learning algorithm, that's again, this this way in a sneaky backdoor way. Or you can say, well, the type of learning for the reward maximization is a different one than the learning we mean here, here, we mean the acquisition of knowledge, but I'm pretty sure the acquisition of knowledge is part of reward maximization. So a little bit of a close loop there. Honestly. Yeah. So I'm not I'm not sure. But here they make the case and of course, like I agree with all of this, I agree that RL, you know, reward maximization, if you have a powerful enough algorithm, it will sort of discover these sub tasks and it will has to acquire these abilities and so on, it might not be super sample efficient. And certainly, it's a better way to general and to general intelligence than like supervised learning or or just prediction itself, like future prediction and so on. That is, and that online learning is better than offline learning. I agree with all of this, right. And here in the discussion, by the way, they also say which environment, right, and then they say, well, it can be any as long as it's, I guess, complex enough, which reward signal and here they also they give this this pebble example, where they say, well, even a simple goal in the complex environment can can give rise or will give rise to intelligence. And yeah, so I agree with most of it. But this this core, the core thing, I'm just very skeptical about. And my last comment here is that they, they so presuppose that this reward exists, right, they so presuppose that there is an environment that gives you a reward. And I also disagree with that, right. So if you look at the only process that we know that actually has produced artificial or not artificial, natural general intelligence, or at least human like intelligence is evolution, the evolutionary process has given rise to all the intelligence that we see, be that in humans or other animals today, or, or even like social systems, if you consider them to be intelligent corporations, the evolutionary process is the main backbone of this. And even though you can maybe formulate the evolutionary process as maximizing fitness, I don't like there is no for evolution, there is. I don't think that's a good way of framing it. It's rather that there is an absence of reward. And it is not a single reward that's given, right, that the reward creates itself as evolution goes into a new niche. And it is not a, a, you can consider it in the abstract as a reward maximization algorithm, if you consider fitness to be your reward. But I do, I do. In my mind, there is a difference between this and what these people have in mind right here of giving rise to social or to intelligence in that I would consider the evolutionary method to have no reward in the sense that they describe it right here. And yeah, so that is, to me, the the last sort of the kind of more baseline flaw in this paper, in that it assumes that there even is such a thing as a reward in the environment, where I'm absolutely not sure that this is even the case. And the only example that we have of general, or at least good intelligence, human like intelligence, however you want to call it, is a process that arguably has no reward. And it just does what works. And some of the stuff that works is intelligent. But other stuff with the exact same apparent reward signal is absolutely not intelligent. And that leads me to believe to say, No, I don't think reward is, as you say, enough. But that's my opinion. Please give me your opinion in the comments. I still read the comments. And I will do so in the future. So if you're watching this in 2040, and if you're an AI, then please write in the comment, did you did you just use reward? Or did you use something else? I'd be very interested. Also, please, please spare me. Yeah, I'll see you next time. If I'm still here. Bye bye.
 From the makers of Is All You Need and Do We Really Need and Is It Even Useful now comes
 Enough.
 So, today we're going to look at Reward Is Enough by David Silver, Satinder Singh, Doina
 Prakup and Richard S. Sutton.
 This paper is a more philosophical paper, I feel, though it presents itself as having
 practical advice in it.
 And the core hypothesis in this paper, and they state it as a hypothesis, is that maximizing
 reward in a sufficiently complex environment is a sufficient condition for intelligence
 to arise implicitly in service of maximizing that reward.
 So the example they give is like a squirrel who wants to get as many nuts as possible,
 has to learn to do all kinds of things in the environment.
 In order to do that, it needs to know how to perceive, how to motor act in the world,
 it needs to understand maybe the cycles of the year, it needs to be able to communicate
 and fend away other squirrels and so on.
 So a lot of these abilities naturally arise from something that just wants to maximize
 a reward in a complex environment.
 I do have my troubles with this hypothesis right here, especially how they present it,
 but we'll go through the paper, look at the hypothesis, at the reasoning, and as always,
 tell me what you think about this work.
 The conclusion of the work is that if this is correct, this sort of gives a straight
 path to general intelligence, namely, let's just maximize reward in a sufficiently complex
 environment.
 And as always, if you do like it, share it out, subscribe if you haven't, and we'll dive
 into the paper.
 So the abstract says, in this article, we hypothesize that intelligence and its associated
 abilities can be understood as subserving the maximization of reward.
 Accordingly, reward is enough to drive behavior that exhibits abilities studied in natural
 and artificial intelligence, including knowledge, learning, perception, social intelligence,
 language, generalization, and imitation.
 This is in contrast to the view that specialized problem formulations are needed for each ability
 based on other signals or objectives.
 Furthermore, we suggest that agents learn through trial and error experience to maximize
 reward could learn behavior that exhibits most if not all of these abilities.
 So it's agents that learn through trial and error.
 And therefore, that powerful reinforcement learning agents could constitute a solution
 to artificial general intelligence.
 Now this has sort of this is kind of the deep mind ethos, right in a nutshell, it is let's
 just build in not like the most powerful reward maximization agents specifically through reinforcement
 learning that we can, and that will sort of get us to general intelligence because in
 order to achieve anything in the world, you need to be intelligent if you want to achieve
 it to a very, very high degree.
 Now if that tickles you a bit in the wrong spot, so it does the same to me.
 But so they contrast this here.
 They ask how does intelligent intelligence arise?
 How does it arise?
 And how is it so bountiful and so varied and has very different subsystems?
 And how does this come about?
 They say one possible answer is that each ability arises from the pursuit of a goal
 that is designed specifically to elicit that ability.
 So for example, the ability of social intelligence has often been framed as the Nash equilibrium
 of a multi agent system.
 And they go through others.
 In this paper, they say we consider an alternative hypothesis that the generic objective of maximizing
 reward is enough to drive behavior that exhibits most if not all abilities that are studied
 in natural and artificial intelligence.
 So they give an example right here with the with the squirrel.
 And so one example is a squirrel in sort of the natural world.
 And the other example is a kitchen robot or a household robot, also in the natural world.
 Now the one of the core points of this paper is that the environment needs to be let's
 say, complex enough.
 And I feel like they're only going to be satisfied with a particular environment than that is
 the real world.
 So if they say a complex environment, just think of the real world, like be that, you
 know, agents on the real internet in the real world, or be that squirrels in the actual
 physical world, they think of environments that are sufficiently complex.
 And that's sort of how this hypothesis draws their power.
 So the description of this figure says, the reward is enough hypothesis postulates that
 intelligence yada yada yada.
 For example, a squirrel acts as to maximize its consumption of food, that's the at the
 top right here, which is the reward depicted by the acorn the acorn symbol, or a kitchen
 robot acts as to maximize cleanliness.
 To achieve these goals, complex behaviors are required that exhibit a wide variety of
 abilities associated with intelligence.
 Okay, so the squirrel must learn to perceive it must learn to climb, it must learn to assess
 the knots, it must learn to bury them, it must learn to remember where they are, and
 so on.
 And the cleanliness robot must learn also to perceive to use its sort of movements,
 it must learn to wash.
 And it might even decide, let's get pizza delivered instead of instead of cooking, because
 that will be just cleaner, arguable.
 But yeah, so in in this framework, you can see on the right here, they see all of these
 different abilities, such as memory, perception, planning, and so on, just arising from these
 things, because they say, well, in order for the squirrel to maximize nuts, it needs to
 be able to do all of these things, otherwise, the squirrel will just sort of die.
 It can't, it can't, like without perceiving the nuts, it can't go get the nuts.
 And the also the cleanliness robot, if it is actually good at maximizing its reward,
 it needs to develop all these abilities, including right, like the social abilities in order
 to get a pizza delivered or in order to work together with the human, maybe even to manipulate
 the human to make less dirt.
 So that's the that's essentially the hypothesis right here.
 They do give some example.
 So they I mean, this first part, the introduction, I mean, you can read it for yourself, but
 they they say, they give these examples here, they say, watching this through the lens of
 reward maximization may, in fact, provide a deeper understanding since it explains why
 such ability arises, for example, avoidance of crocodiles, because you need you don't
 want to be eaten.
 In contrast, when each ability is understood as the solution to its own specialized goals,
 the why question is sidestepped in order to focus upon the what the ability does.
 Singular goal may provide a broader understanding.
 And it might even lead to new sort of new forms of intelligence.
 They give examples, of course, here, the games of go and chess, where just maximizing the
 reward alpha zero was able to come up with very new, very new tactics, very new openings
 and games and so on.
 And we didn't teach it to do openings, we didn't teach it to do board control and whatnot,
 or whatever they call in the things in go, we just asked it to maximize reward.
 And it came up with all of these sort of sub abilities by itself, right?
 Now they formalize this here, the reinforcement learning problem, they formalize it as an
 agent interacting with the environment.
 So here, the agent is just the decision making process.
 So in the squirrel, actually, only the squirrel brain would be the agent and the squirrel
 body is already part of the environment.
 So if you're in a sort of multi agent system, all the other agents are part of the environment
 in this framework.
 And the environment, you interact with it, and you get a reward signal, right?
 Reward signal, and then maximizing that reward signal, that is what you call reward maximization.
 And the core hypothesis of this paper, as I already said, right here, is the reward
 is enough hypothesis.
 And the hypothesis itself says, intelligence and its associated abilities can be understood
 as subserving the maximization of reward by an agent acting in its environment.
 It's a bit better stated above, I think that they say that the main different forms of
 intelligence can be understood as subserving the maximization of reward and that the many
 abilities associated with each each form of intelligence may arise implicitly from the
 pursuit of those rewards taken to its limit, we hypothesize that all intelligence and associated
 abilities may be understood in this manner.
 Now they do strengthen it.
 They do strengthen this hypothesis, because what you might be thinking of what I was thinking
 of first is that, oh, you know, you can just formulate any goal as reward.
 And that's what they say here, they say the reward hypothesis, which is different from
 their hypothesis, speculates that all goals of interest in studying natural or building
 artificial agents may be represented by rewards.
 This should not be confused with our reward is enough hypothesis, which considers the
 abilities that arise from the pursuit of any such any one such goal.
 Okay, so it's different than just saying, well, you can learn to perceive by doing reinforcement
 learning or well, you can learn to acquire knowledge by reinforcement learning.
 Now this is stronger.
 This says that the hypothesis here is intended to be much stronger, that intelligence and
 associated abilities will implicitly arise in the service of maximizing one of many possible
 reward signals corresponding to the many pragmatic goals towards which natural or artificial
 intelligence may be directed.
 So their idea is that there is a world, and that world is sort of complex enough, right?
 Maybe there's a tree, and you know, there's a house, so there is humans in it.
 And you have your little squirrel, whatever here, squirrel has a bushy tail and a head.
 I don't I don't know how squirrel looks just this is a head.
 And given in this environment, you pick any reward you can think of like any any reward
 signal, and then maximize such as like how many how much hunger do you have, you get
 that as a negative reward, and then maximizing that reward will lead implicitly to the squirrel
 having to develop intelligence having to develop perception having to develop the acquisition
 of knowledge, and even interacting with other squirrels or the humans in this world.
 This is a strong hypothesis.
 And as I said, I do have my problems with it.
 First though, they go through a bunch of things they say, well, let's explore how we let's
 explore some abilities that people naturally associate with intelligence.
 And let's explore how they might arise implicitly from reward maximization.
 Okay, so again, think of the squirrel wanting to get as many nuts as possible, or like,
 I don't know, a human wanting to survive and live and thrive in the real world, how something
 like intelligence may arise just as a product of maximizing that reward.
 And so here they go over a bunch of them.
 The first one is knowledge and learning.
 And the the arguments made here are always they're always pretty simple.
 They're they're giving you an example and saying, well, in order to maximize your reward
 in the real world, it's useful to have knowledge.
 And also because you don't have infinite memory or whatnot, it's useful to learn things and
 to abstract things right to to gather knowledge and so on.
 And then when here when they go for perception, they say, well, in order to maximize your
 reward to thrive, you need to perceive.
 Okay, so, you know, naturally, it's like almost a tautology.
 Okay, so they say, well, a reward maximization agent can reward maximize better if it perceives
 rather than if it doesn't perceive.
 Okay, so it's, it's, it's sort of and social intelligence.
 Yes.
 So if you're a human, you want to thrive in the world, it's better if you are socially
 intelligent.
 In fact, it's better if you know language because you can maximize reward by communicating.
 So language, if if you know might just be a byproduct of reward maximization, generalization.
 Well, it's better if you generalize and imitation.
 Yes, it's better if you imitate general intelligence.
 Well, if you want to reward maximize, you need to be able to instant sort of switch
 around between different sub goals in order to reward maximize and sort of solve new problems
 really easily.
 That would be really good in order for you to maximize your reward.
 And therefore general intelligence is might be, you know, if an if an agent maximized
 its reward, general intelligence will help.
 And I hope you've seen a little bit the trend here through all of these things.
 And I think especially in the last thing, in this general intelligence, the the flaw
 here, what I think is the flaw becomes rather obvious, because I mean, so reward is enough
 for for general intelligence.
 Essentially, you're saying, well, if we build something that's intelligent, right, then
 we have then intelligence is a byproduct of that.
 So if if you if you postulate your reward maximization as being intelligent, then yes,
 intelligence arises as a byproduct.
 Their their whole notion here is that if you have this complex environment, and you want
 to do anything, you need to be intelligent.
 And that's how they see the environment itself.
 The big question here is, of course, what is this environment?
 And what is the reward?
 And they have a discussion at the end where they say, well, as long as the environment
 is complex enough, we don't actually care, right?
 If it's complex enough, you know, the any and also for the reward, like any reward signal,
 any goal will do you can and they say, well, what if you if you're if your goal is to collect
 pebbles in the real world?
 Okay, so, you know, there is a pebble.
 There is a pebble.
 There is a pebble.
 So one agent might just learn to collect pebbles.
 But the other agent might learn to sort of use the internet and buy pebble collectors
 off of Amazon and then launch a political campaign and influence all the humans to also
 collect pebbles for itself and then influence everything and get rich and buy more pebbles.
 And that would necessitate intelligence.
 So just maximizing getting pebbles would sort of lead to intelligence.
 And I'm, I follow this way.
 But you know, again, this is sort of saying, if you're intelligent, then you're intelligent.
 And on the other hand, what if a agent could simply chemically transform anything it finds
 into pebbles or anything that's even possible?
 There's this this meme, right with the distribution, where here is the new guy.
 So here you have like, here we have this guy with this hair and with the teeth and this
 goes collect collect pebbles.
 And then here you have the I don't know, here's the smart person usually.
 And this person is like, well, influence all the people and buy things with money and do
 this and do that and do this and do that.
 And over here, I just imagine the the Zen.
 So there's usually the the person in the hoodie, right?
 The Zen person.
 Well, that's a terrible hoodie.
 The Zen person again going collect pebbles.
 Like you don't know this.
 I think this is such a this is such it's just kind of looking out at the world and then
 abstracting that into what they consider a reward of the environment.
 And then naturally tautologically, what will arise is that if you sort of maximize that,
 then intelligence will arise.
 And that's not even the end of it, right?
 Because a lot of things such as survival in the world and thriving in different environments
 are done without intelligence.
 If you think of bacteria, for example, bacteria, so I don't know.
 So here's the world.
 And there's like a tiny sliver where humans can live in about one fourth or so of that
 sliver.
 Yet bacteria, they're everywhere.
 Okay, they thrive much more than humans.
 So if the if the goal is survival and fitness, I mean, bacteria solve that problem completely
 without any intelligence.
 So I disagree that just reward maximization is enough.
 But then these people would say, well, the environment is not the same.
 The environment for a bacteria is not the same as for a human.
 Like if you are a human, clearly, your approach cannot be to just replicate.
 So if you're a bacteria, you know, here is here your bacteria, what do you do?
 You simply split.
 Cool.
 Don't need intelligence can colonize the entire planet.
 However, if you're a human, that is not an option.
 If you're a human, you need to be intelligent, right?
 Your environment is different.
 So your environment is much more what they would say complex, though I disagree, I think
 that bacteria's environment is incredibly complex.
 But the human environment, they would say is so complex.
 You as a human need intelligence in order to thrive that environment.
 Now again, there is a fallacy here, in my opinion, right in my opinion, what do I know?
 This is rich Sutton.
 But in my opinion, there is a fallacy here, namely, so there is the environment, right?
 And you're you're the human right here, you're in the environment.
 And in order to maximize your reward as a human, because you can't split because there
 are other humans around, you need intelligence, right?
 Intelligence needs to be right here in the human in order to survive and thrive in the
 human environment.
 However, that environment only exists because there is already intelligence, right?
 So first of all, you as a human, you don't acquire intelligence because you need it in
 your environment, you have it built into you.
 You do a bit of fine tuning during your life, but not like the no one doubts that a that
 intelligence is present even in a baby, okay, like it might not be able to, to act it out.
 But the all of the ingredients like the learning, the the ability to absorb knowledge and so
 on that like the ability to perceive and to to learn language that is all present already.
 So I disagree that humans acquire and have to acquire intelligence in order to thrive.
 Now they people would say, well, evolution, the evolutionary pressure on humans required
 intelligence and that might be true.
 But the individual human only needs intelligence because intelligence is already present in
 the environment, or if you want to call it differently.
 So here is your world and you can go into different niches, right?
 And one of the niches is the bacteria niche where you simply you simply split.
 Okay, another niche, another environmental niche is the niche where in fact you need
 intelligence in order to survive.
 But that is determined.
 That is just this niche, right?
 And you need intelligence because the other humans have intelligence.
 And because you were you're only born as a human, because you're because the the environment
 has or the evolutionary direction has pushed you into that direction.
 So it is not that the maximization of any reward be that fitness has led to intelligence
 because the maximization of that same reward has also not led to intelligence.
 It's simply that intelligence is present in this particular niche of the evolutionary
 process.
 Right?
 I see this as a clear distinction.
 Like I feel humans first of all, they have innate intelligence.
 And second of all, the environment is only such that intelligence is necessary because
 other humans before you also had intelligence.
 Nowhere in this process is the environment determinist or the driver of the development
 of intelligence.
 Because at the beginning, right here, the environment wasn't such that intelligence
 was necessary.
 So the environments and the intelligence they evolve together, sorry, the the environment
 that requires intelligence and the intelligent beings evolve together.
 At no point did you have an environment that required intelligence because of maximization
 of reward.
 And you had an object in that environment, not having intelligence and then having to
 acquire it.
 It's simply one niche.
 And there are other niches that don't require it.
 So that's, that's, that's my one of the largest things that I criticize right here, I disagree
 that reward maximization is enough for intelligence, because clearly the same reward maximization
 wasn't enough in other cases.
 Also, I think that there is no such like if they think of the real world, and agents with
 intelligence in it, those agents only exist because intelligence exists, not the other
 way around.
 The agents don't make intelligence, they already are intelligent for the most part.
 And the last thing right here is, I just want to point to you here that reward is enough
 for knowledge and learning.
 So now, they call learning one of these abilities that is associated with intelligence.
 And now we go to the next part.
 And the next part is where they ask themselves, well, given that we postulate that maximizing
 reward might be enough for intelligence, how should we achieve that?
 So it the hypothesis of maximization of reward is fully agnostic to the nature of the agent
 itself.
 This leaves open the important question on how to construct an agent that maximizes reward.
 So that's the question, right?
 How do you construct an agent that maximizes reward?
 Until now, we've heard no, of course, the answer is going to be reinforcement learning.
 But until now, we have actually not heard much of that except in examples.
 So they still leave it open how you would achieve such an agent.
 But now they're going to say reinforcement learning.
 But first they say, in this section, we suggest that this question may also be largely answered
 by reward maximization.
 Now I don't actually know whether this is intended here.
 But how to construct an agent that maximizes reward is largely answered by reward maximization.
 Like is this intended?
 Is this an intended back reference saying like, how do we construct x?
 Well x, like, is this, I'm not sure.
 Is this an intended like a little bit of a slight of like a little bit of a joke or something?
 I'm not sure.
 I'm not sure.
 I might just be too dumb, right?
 Specifically, we consider agents with the general ability to learn how to maximize the
 reward from their ongoing experience of interacting with the environment.
 Such agents we will refer to as reinforcement learning agents provide several advantages.
 So here they go into, you know, if you don't want to pre program, like you don't want to
 have the designer's knowledge of the environment be in there because the designer doesn't know
 everything, you want to actually let the agents learn themselves.
 And if the environment is sufficiently complex, and the reinforcement learning agent is sufficiently
 powerful, then it will like the richness of experience of a complex environment will provide
 enough signal for the agent, you know, disregard its practical implementation and sample complexity.
 Technically, the whole the whole richness of experience will provide enough of a signal
 to learn all of this.
 But I don't know, did you?
 There's another thing right here.
 We consider agents with a general ability to learn how to maximize reward.
 So how do we build reward maximization agents, which if successful will give rise to intelligence?
 Right?
 Well, by learning, okay.
 However, learning up here, learning is a product of intelligence or an ability that comes with
 intelligence, right?
 So like we need, we need learning in like learning comes with Intel learning is one
 of the abilities that indicates intelligence.
 So a little bit, it's like learning gens.
 So intelligence, if something is intelligent, right, it then then it will learn but also
 in order to achieve this intelligence through reward maximization, that's how we achieve
 intelligence but then in order to do reward maximization, we need a learning algorithm.
 But if the learning algorithm is not yet intelligent, right, then how is this happening?
 So I think you can I guess you can make a split and saying, well, this learning that
 we we use for reward maximization, that's sort of a learning that we design or something
 like this.
 But even if we design it, intelligence gives like if we design the learning algorithm,
 that's again, this this way in a sneaky backdoor way.
 Or you can say, well, the type of learning for the reward maximization is a different
 one than the learning we mean here, here, we mean the acquisition of knowledge, but
 I'm pretty sure the acquisition of knowledge is part of reward maximization.
 So a little bit of a close loop there.
 Honestly.
 Yeah.
 So I'm not I'm not sure.
 But here they make the case and of course, like I agree with all of this, I agree that
 RL, you know, reward maximization, if you have a powerful enough algorithm, it will
 sort of discover these sub tasks and it will has to acquire these abilities and so on,
 it might not be super sample efficient.
 And certainly, it's a better way to general and to general intelligence than like supervised
 learning or or just prediction itself, like future prediction and so on.
 That is, and that online learning is better than offline learning.
 I agree with all of this, right.
 And here in the discussion, by the way, they also say which environment, right, and then
 they say, well, it can be any as long as it's, I guess, complex enough, which reward signal
 and here they also they give this this pebble example, where they say, well, even a simple
 goal in the complex environment can can give rise or will give rise to intelligence.
 And yeah, so I agree with most of it.
 But this this core, the core thing, I'm just very skeptical about.
 And my last comment here is that they, they so presuppose that this reward exists, right,
 they so presuppose that there is an environment that gives you a reward.
 And I also disagree with that, right.
 So if you look at the only process that we know that actually has produced artificial
 or not artificial, natural general intelligence, or at least human like intelligence is evolution,
 the evolutionary process has given rise to all the intelligence that we see, be that
 in humans or other animals today, or, or even like social systems, if you consider them
 to be intelligent corporations, the evolutionary process is the main backbone of this.
 And even though you can maybe formulate the evolutionary process as maximizing fitness,
 I don't like there is no for evolution, there is.
 I don't think that's a good way of framing it.
 It's rather that there is an absence of reward. And it is not a single reward that's given,
 right, that the reward creates itself as evolution goes into a new niche.
 And it is not a, a, you can consider it in the abstract as a reward maximization algorithm,
 if you consider fitness to be your reward. But I do, I do.
 In my mind, there is a difference between this and what these people have in mind right
 here of giving rise to social or to intelligence in that I would consider the evolutionary
 method to have no reward in the sense that they describe it right here.
 And yeah, so that is, to me, the the last sort of the kind of more baseline flaw in
 this paper, in that it assumes that there even is such a thing as a reward in the environment,
 where I'm absolutely not sure that this is even the case.
 And the only example that we have of general, or at least good intelligence, human like
 intelligence, however you want to call it, is a process that arguably has no reward.
 And it just does what works. And some of the stuff that works is intelligent. But other
 stuff with the exact same apparent reward signal is absolutely not intelligent. And
 that leads me to believe to say, No, I don't think reward is, as you say, enough.
 But that's my opinion. Please give me your opinion in the comments. I still read the
 comments. And I will do so in the future. So if you're watching this in 2040, and if
 you're an AI, then please write in the comment, did you did you just use reward? Or did you
 use something else? I'd be very interested. Also, please, please spare me. Yeah, I'll
 see you next time. If I'm still here. Bye bye.