 Hi. This video is a recording of a talk that I gave a while back. I already published a version of it on my second channel, but did you even know I had a second channel? Most people don't. I thought more people should see it, so I remastered it. I cleaned it up, improved the graphics, and yeah, this is that. Enjoy. Right. Hello everyone. My name is Robert Miles. I am usually doing this on YouTube. I'm not really used to public speaking. I'm not used to not being able to edit out my mistakes. There may be mistakes. Also, I may go too quickly. Sorry, not sorry. So when it comes to AI safety, you can kind of divide it up into four areas along two axes. You've got your short term and your long term, and you've got accident risks and misuse risks. And that's kind of a useful way to divide things up. AI safety covers everything. The area that interests me most is the long-term accident risks. I think once you have very powerful AI systems, it almost doesn't matter if they're being used by the right people or the wrong people or what you're trying to do with them the difficulty is in keeping them in under control at all so that's what i'm going to be talking about what is ai safety why is it important so i want to start by asking the question which i think everybody needs to be asking themselves what is the most important problem in your field take a second to think of it and why are you not working on that? For me I think the most important problem in the field of AI is AI safety. This is the problem specifically that I'm worried about. That we will sooner or later build an artificial agent with general intelligence. So I'm going to go into a bunch of these terms. The first thing is what do I mean when I say sooner or later? This is a little bit washed out but this is a graph of a survey, a large survey of AI experts. These are people who published in major AI conferences and they were asked when they thought we would achieve high-level machine intelligence, which is defined as an agent which is able to carry out any task humans can as well as or better than humans. And they say that a 50% chance of having achieved that, we hit that about 45 years from 2016. but then of course we hit like 10% chance nine years from now so it's not immediate but it's happening this is definitely worth taking with a pinch of salt because if you ask the question slightly differently you get an estimate of 120 years rather than 45 there's a lot of uncertainty in this area. But the point is, it's going to happen, as I said, sooner or later. Because at the end of the day, general intelligence is possible. The brain implements it. And the brain is not magic. Sooner or later, we'll figure it out. So what do I mean when I say an artificial agent? Well, so an agent is a term from economics mostly. But basically agents have goals, they choose actions to further their goals. That is the simplest expression of what an agent is. So the simplest thing that you might call an agent would be something like a thermostat. It has a goal which is to have the room be at a particular temperature. It has actions it can take, it can turn on the heating. It can turn on the air conditioning. It chooses its actions to achieve its goal of maintaining the room at a steady temperature. Extremely simple agent. A more complex agent might be something like a chess AI, which has a goal of, like if it's playing white, it has a goal of the black king being in checkmate. And it takes actions in the form of moving pieces on the board in order to achieve its goal. So you see how this idea of an agent is a very useful way of thinking about lots of different intelligent systems. And of course, humans can be modeled as agents as well. This is how it's usually done in economics. Individuals or companies could be considered to have a goal of maximizing their income or maximizing their profits and making decisions in order to achieve that. So when I'm talking about intelligence, intelligence has a lot of, as a term, is a heavily loaded term, has a lot of different people put their own definitions on it. In this context, what I mean when I say intelligence is just the thing that lets an agent choose effective actions. It's whatever it is that's in our brains or that's in the programming of these systems that means that the actions they choose tend to get them closer to their goals. And so then you could say that an agent is more intelligent if it's more effective at achieving its goals, whatever those goals are. If you have two agents in an environment with incompatible goals, like let's say the environment is the chessboard, and one agent wants white to win, and one agent wants black to win, then generally the more intelligent agent will be the one that gets what it wants. The better AI will win the chess game. And finally, general intelligence. This is where it becomes interesting in my opinion. So generality is the ability to behave intelligently in a wide range of domains. If you take something like a chess AI, it's extremely narrow. It only knows how to play chess. And even though you might say that it's more intelligent than a thermostat because it's more sophisticated, it's more complicated, it couldn't do the thermostat's job. There's no position on the chessboard that corresponds to the room being a good temperature. There's no move that corresponds to turning on an air conditioner. The chess AI can only think in terms of chess. It's extremely narrow. Generality is a continuous spectrum. So if you write a program that can play an Atari game, that's very narrow. DeepMind, one of their early triumphs was that they made a program that could play dozens of different Atari games. A single program that could learn all of these different games. And so it's more general. Because it's able to act across a wider variety of domains. The most general intelligence that we're aware of right now is human beings. Human beings are very general. We're able to operate across a very wide range of domains, including, and this is important, we're able to learn domains which evolution did not and could not prepare us for. We can, for example, drive a car, and evolution did not prepare us for that We can, for example, drive a car. And evolution did not prepare us for that. We invented cars. They're very recent. We can, you know, invent rockets and go to the moon. And then we can operate on the moon, which is a completely different environment. And this is kind of the power of general intelligence. Really, the power of general intelligence is we can build a car, we can build a rocket, we can put the car on the rocket, take the car to the moon, drive the car on the moon. And there's nothing else that can do that yet, but sooner or later, right? So this is what I'm talking about. I'm talking about what you might call true AI, real AI, the sci-fi stuff. An agent which has goals in the real world and is able to intelligently choose actions in the real world to achieve those goals. Now that sounds... I've said, what's the biggest problem? This doesn't sound like a problem, right? On the surface of it, this sounds like a solution. You just tell the thing, you know, cure cancer or maximize the profits of my company or whatever. And it takes whatever actions are necessary in the real world to achieve that goal. But it is a problem. So the big problem is, this should be auto-playing and it isn't. The big problem is it's difficult to choose good goals. So this is an AI made by OpenAI. It's playing a game called Coast Runners, which is actually a racing game. They trained it on the score, which you probably can't see down here. It's currently a thousand. What the system learned is that if it goes around in a circle here and crashes into everything and catches fire, these little turbo pickups, they respawn at just the right rate. But if it just flings itself around in a circle, it can pick up the turbo. And that gives you a few points every time you do that. And it turns out that this is a much better way of getting points than actually racing around the track. And the important point here is that this is not unusual. This is not OpenAI doing anything unusually stupid. This is kind of the default. Picking objectives is surprisingly hard. And you will find that the strategy or the behavior that maximizes your objective is probably not the thing you thought it was. It's probably not what you were aiming for. There's loads of examples. Actually, Victoria has a great list on her blog, Deep Safety, there's like 30 of them, different things going wrong. There was one they had, they were trying to teach, they were trying to evolve systems that would run quickly, so they trained them on the, I'm gonna pause this because it's distracting as hell. Where's my mouse? They're mesmerizing. Yeah. Pause, pause, please. They were training agents that were supposed to run. So they simulated them for a particular period of time and measured how far their center of mass moved, which seems perfectly sensible. What they found was that they developed a bunch of these creatures, which were extremely tall and thin with a big mass on the end, that then fell over. Because they weren't simulating them for long enough that you could go the fastest just by falling over rather than actually running. That moved your center of mass the furthest. There's a lot of these. There was a Tetris bot which would play reasonably well, and then just when it was about to lose, would pause the game and sit there indefinitely. Because it lost points for losing, but didn't lose any points for just sitting on the pause screen indefinitely. This is like the default of how these systems behave. I have no memory what my next slide is. Oh yeah, right. So we have problems specifying even simple goals in simple environments like Atari games or basic evolutionary algorithms, things like that. When it comes to the real world, things get way more complicated. This is a quote from Stuart Russell, who sort of wrote the book on AI. When a system is optimizing a function of n variables where the objective depends on a subset of size k, which is less than n, it will often set the remaining unconstrained variables to extreme values. If one of those unconstrained variables is something that we care about, the solution found may be highly undesirable. In the real world, we have a very large number of variables, and so we're talking about very large values for n here. So let's say you've got your robot, and you've given it a goal which you think is very simple. You want it to get you a cup of tea. So you've managed to specify what a cup of tea is and that you want one to be on the desk in front of you. So far, so good. But suppose there's a price-listening vase on a narrow stand in front of where the kitchen is. So the robot immediately plows into the vase and destroys it on its way to make you a cup of tea, because you only gave it one variable to keep track of in the goal, which is the T. It doesn't care about the vars. You never told it to care about the vars. It destroys the vars. This is a problem. So, okay, now you can shut it down, modify it, and say, okay, get me a cup of tea, but also don't knock over the vars. But then there will be a third thing. There is always another thing. Because when you're making decisions in the real world, you're always making trade-offs. You're always taking various things that you value and deciding how much of one you're willing to trade for how much of another. You know, I could do this quicker, but it increases the risk of me making a mistake. Or I could do this cheaper, but it won't be as reliable. I could do this quicker, but it increases the risk of me making a mistake. Or I could do this cheaper, but it won't be as reliable. I could do this faster, but it'll be more expensive. You're always trading these things off against one another. And so an agent like this, which only cares about a limited subset of the variables in the system, will be willing to trade off arbitrarily large amounts of any of the variables that aren't part of its goal for arbitrarily tiny increases in any of the things which are in its goal. So it will happily... Let's say now, for example, now it values the vase, and those are the only things that it values. It might reason something like, OK, there's a human in the environment. The human moves around. The human may accidentally knock over the vase, and I care about the vase, so I have to kill the human. And this is totally ridiculous, but if you didn't tell it that you value being alive, it doesn't care. And anything that it doesn't value is going to be lost. If you have a sufficiently powerful agent, and you manage to come up with a really good objective function which covers the top 20 things that humans value. The 21st thing that humans value is probably gone forever, because the smarter, the more powerful the agent is, the better it will be at figuring out ways to make these trade-offs, to gain a millionth of a percent better at one thing while sacrificing everything of some other variable. So this is a problem. But actually, that scenario I gave was unrealistic in many ways. But one important way that it was unrealistic is that I had the system go wrong, and then you just turn it off and fix it. But in fact, if the thing has a goal of getting you a cup of tea, this is not like a chess AI, where you can just turn it off because it has no concept of itself or of being turned off. Its world model contains you, it contains itself, it contains the possibility of being turned off, and it's fully aware that if you turn it off, because it knocked over the vase, it won't be able to get you any tea, which is the only thing it cares about. So it's not going to just let you turn it off. It will fight you. Or if it's slightly smarter, it will deceive you so that you believe it's working correctly, so that you don't want to change it until it's in a position where you can't turn it off. And then it will go after its actual objective. So this is a problem. And the thing is, this is a convergent instrumental goal, which means it sort of doesn't matter what the goal is. It doesn't matter what your goal is as an agent. If you're destroyed, you can't achieve that goal. So it sort of almost doesn't matter what goal we give it. There is only a very tiny fraction of possible goals that will involve it actually allowing itself to be turned off and modified. And that's quite complicated. There are some other convergent instrumental goals. So we had self-preservation, goal preservation. Resource acquisition is the kind of thing we can expect these kinds of systems to do. Most plans, you can do them better if you have more resources, whether that's money, computational resources, just free energy, matter, whatever. The other one is self-improvement. Whatever you're trying to do, you can probably do it better if you're smarter. And AI systems potentially have the capacity to improve themselves, either just by acquiring more hardware to run on or changing you know improving their their software to run faster or better or so on so there's a whole bunch of behaviors which intelligent systems intelligent agents generally intelligent agents we would expect them to do by default. And that's really my core point. Artificial general intelligence is dangerous by default. It's much, much easier to build these kinds of agents which try to do ridiculous things and trick you and try to deceive you or will fight you when you try to turn them off or modify them on the way to doing some ridiculous thing which you don't want, much easier to build that kind of agent than it is to build something which actually reliably does what you want it to do. And that's why we have a problem, because we have 45 to 120 years to figure out how to do it safely, which is a much harder problem. And we may only get one shot. It's entirely possible that the first true artificial general intelligence will manage to successfully achieve whatever its stupid goal is. And that could be truly a disaster on a global scale. So we have to beat this challenge on hard mode before anyone beats it on easy mode. So are we screwed? No. We're only probably screwed. there are things we can do safe general artificial intelligence is totally possible it's just a very difficult technical challenge and there are people working very hard on it right now trying to solve a whole range of difficult technical challenges so that we can figure out how to do this safely thanks of difficult technical challenges so that we can figure out how to do this safely. Thanks. You may have noticed in the intro and this outro that the image quality has improved since the last video. That's largely thanks to my excellent patrons. Thank you to all of these people here for helping me to get this new camera. In this video, I'm especially thanking James Petz, who's been hanging out with us on the Discord server, helping answer questions from the YouTube comments and so on. And actually, that last video about Mesa optimizers has had a lot of really good questions. So the next video, we'll be answering some of those. That's coming out soon. So thanks again to James and to all my patrons, to everyone who asked questions, so the next video will be answering some of those. That's coming up soon. So thanks again to James and to all my patrons, to everyone who asked questions, and to you for watching. I'll see you next time.
Intro to AI Safety, Remastered
An introduction to AI Safety, remastered from a talk I gave at "AI and Politics" in London

The second channel: https://www.youtube.com/channel/UC4qH2AHly_RSRze1bUqSSNw

Experts' Predictions about the Future of AI: http://youtu.be/HOJ1NVtlnyQ
9 Examples of Specification Gaming: http://youtu.be/nKJlF-olKmg

https://www.patreon.com/robertskmiles
With thanks to my wonderful Patreon supporters:
Gladamas
Timothy Lillicrap
Kieryn 
AxisAngles 
James 
Nestor Politics
Scott Worley
James Kirkland
James E. Petts
Chad Jones
Shevis Johnson
JJ Hepboin
Pedro A Ortega
Said Polat
Chris Canal
Jake Ehrlich
Kellen lask
Francisco Tolmasky
Michael Andregg
David Reid
Peter Rolf
Teague Lasser
Andrew Blackledge
Frank Marsman
Brad Brookshire
Cam MacFarlane
Craig Mederios
Jon Wright
CaptObvious
Brian Lonergan
Jason Hise
Phil Moyer
Erik de Bruijn
Alec Johnson
Clemens Arbesser
Ludwig Schubert
Eric James
Matheson Bayley
Qeith Wreid
jugettje dutchking
Owen Campbell-Moore
Atzin Espino-Murnane
Johnny Vaughan
Carsten Milkau
Jacob Van Buren
Jonatan R
Ingvi Gautsson
Michael Greve
Tom O'Connor
Laura Olds
Jon Halliday
Paul Hobbs
Jeroen De Dauw
Cooper Lawton
Tim Neilson
Eric Scammell
Igor Keller
Ben Glanton
Tor Barstad
Duncan Orr
Will Glynn
Tyler Herrmann
Ian Munro
Joshua Davis
Jérôme Beaulieu
Nathan Fish
Peter Hozák
Taras Bobrovytsky
Jeremy 
Vaskó Richárd
Benjamin Watkin
Andrew Harcourt
Luc Ritchie
Nicholas Guyett
James Hinchcliffe
12tone 
Oliver Habryka
Chris Beacham
Zachary Gidwitz
Nikita Kiriy
Andrew Schreiber
Steve Trambert
Braden Tisdale
Abigail Novick
Serge Var
Mink
Chris Rimmer
Edmund Fokschaner
J
Nate Gardner
John Aslanides
Mara
ErikBln
DragonSheep 
Richard Newcombe
Joshua Michel
Alex Altair
P 
David Morgan
Fionn 
Dmitri Afanasjev
Marcel Ward
Andrew Weir
Kabs
Ammar Mousali
Miłosz Wierzbicki
Tendayi Mawushe
Jake Fish
Wr4thon 
Martin Ottosen
Robert Hildebrandt
Andy Kobre
Kees 
Darko Sperac
Robert Valdimarsson
loopuleasa
Marco Tiraboschi
Michael Kuhinica
Fraser Cain
Klemen Slavic
Patrick Henderson
Oct todo22
Melisa Kostrzewski
Hendrik 
Daniel Munter
Alex Knauth
Kasper 
Ian Reyes
James Fowkes
Tom Sayer
Len 
Alan Bandurka
Ben H
Simon Pilkington
Daniel Kokotajlo
Yuchong Li
Diagon 
Andreas Blomqvist
Bertalan Bodor
Qwijibo (James)
Zubin Madon
Zannheim 
Daniel Eickhardt
lyon549 
14zRobot 
Ivan 
Jason Cherry
Igor (Kerogi) Kostenko
ib_ 
Thomas Dingemanse
Stuart Alldritt
Alexander Brown
Devon Bernard
Ted Stokes
Jesper Andersson
DeepFriedJif 
Chris Dinant
Raphaël Lévy
Johannes Walter
Matt Stanton
Garrett Maring
Anthony Chiu
Ghaith Tarawneh
Julian Schulz
Stellated Hexahedron
Caleb 
Scott Viteri
Clay Upton
Conor Comiconor
Michael Roeschter
Georg Grass
Isak Renström
Matthias Hölzl
Jim Renney
Edison Franklin
Piers Calderwood
Mikhail Tikhomirov
Matt Brauer
Jaeson Booker
Mateusz Krzaczek
Artem Honcharov
Michael Walters
Tomasz Gliniecki
Mihaly Barasz
Mark Woodward
Ranzear
Neil Palmere
Rajeen Nabid
Christian Epple
Clark Schaefer
Olivier Coutu
Iestyn bleasdale-shepherd
MojoExMachina
Marek Belski
Luke Peterson
Eric Eldard
Eric Rogstad
Eric Carlson
Caleb Larson
Max Chiswick
Aron
Sam Freedo
slindenau
A21 
Johannes Lindmark
Nicholas Turner
Intensifier
Valerio Galieni
FJannis
Grant Parks
Ryan W Ammons
This person's name is too hard to pronounce
kp 
contalloomlegs
Everardo González Ávalos
Knut Løklingholm
Andrew McKnight
Andrei Trifonov
Aleks D
Mutual Information
Tim 
A Socialist Hobgoblin
Bren Ehnebuske
Martin Frassek
Sven Drebitz
https://www.patreon.com/robertskmiles