More Is Different for AI - Scaling Up, Emergence, and Paperclip Maximizers (w/ Jacob Steinhardt)
#ai #interview #research 

Jacob Steinhardt believes that future AI systems will be qualitatively different than the ones we know currently. We talk about how emergence happens when scaling up, what implications that has on AI Safety, and why thought experiments like the Paperclip Maximizer might be more useful than most people think.

OUTLINE:
0:00 Introduction
1:10 Start of Interview
2:10 Blog posts series
3:56 More Is Different for AI (Blog Post)
7:40 Do you think this emergence is mainly a property from the interaction of things?
9:17 How does phase transition or scaling-up play into AI and Machine Learning?
12:10 GPT-3 as an example of qualitative difference in scaling up
14:08 GPT-3 as an emergent phenomenon in context learning
15:58 Brief introduction of different viewpoints on the future of AI and its alignment
18:51 How does the phenomenon of emergence play into this game between the Engineering and the Philosophy viewpoint? 
22:41 Paperclip Maximizer on AI safety and alignment
31:37 Thought Experiments
37:34 Imitative Deception
39:30 TruthfulQA: Measuring How Models Mimic Human Falsehoods (Paper)
42:24 ML Systems Will Have Weird Failure Models (Blog Post)
51:10 Is there any work to get a system to be deceptive?
54:37 Empirical Findings Generalize Surprisingly Far (Blog Post)
1:00:18 What would you recommend to guarantee better AI alignment or safety?
1:05:13 Remarks

References:
https://bounded-regret.ghost.io/more-is-different-for-ai/
https://docs.google.com/document/d/1FbTuRvC4TFWzGYerTKpBU7FJlyvjeOvVYF2uYNFSlOc/edit#heading=h.n1wk9bxo847o

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi, this is an interview with Jacob Steinhardt, who is the author of a blog post series called More is Different for AI. More is Different is the title of a famous paper in science from 1972 by Philip Warren Anderson, a Nobel Prize winner in physics. The article is generally on the theme of emergent phenomenon when scaling things up. So as you make things bigger, not only does stuff get just more as you would expect, but qualitatively new phenomena arise. You know, what better phenomenon to discuss in this context than AI. So today we'll talk to Jacob about this blog post series, expect to learn how scale fundamentally changed how we look at AI systems, how the paperclip maximizer might not be as dumb of a thought experiment, and how we can look forward and make sense of a world where AI safety could play a critical role in how we interact with these systems in the future. Now I'm having a ton of fun talking to people about all kinds of stuff. But ultimately, what matters is you. So please let me know how I can make these videos the best possible for you. Leave a comment, share them around if you like them. And let's get into it. Hello, everyone. Today, I have Jacob Steinhardt here with me who authored a series of blog posts titled More is Different for AI, which lays out an argument or a series of arguments playing out the, I want to say, the different viewpoints on the future of AI alignment and safety in AI safety in machine learning systems, mainly playing on two viewpoints that Jacob calls the engineering viewpoint, mainly focused on, I want to say near term practical things, and the philosophy viewpoint, mainly focused on more overarching principled approaches, but maybe a bit futuristic. And I found this to be super interesting. It's very well laid out. And it also shows a little bit of a journey of Jacob himself, as I think he learned more about these things. So Jacob, thank you very much for being here. Thanks for having me. This was this a was this a an accurate description, let's say of the blog post, there are five in total. How did you come to this? Yeah, I think that's pretty accurate. I'd say the beginning posts, at least are in some sense, almost a kind of letter to my past self, trying to either, you know, argue for for things that I've come to believe now that I didn't believe five years ago, or just viewpoints that I've kind of got more clarity on. And then I think the later posts, start trying to maybe address kind of the broader field. So both, I think I guess you could, I'd say there's maybe two fields that you can think of this as addressing one is the kind of traditional machine learning field, which tends to be very empirically driven. And I wouldn't say is exactly the same as what I'm calling the engineering approach, but I think has a lot of affinity for it. And then this other field, that's kind of more top down, more, more kind of philosophical and conceptual, that's kind of worried about long term risks from AI, that starts with maybe people like Nick Bostrom, who was in fact a philosopher. And so I kind of again, not exactly put that field the same as the philosophy approach, but I think has a lot of affinity for it. And I think my thinking is kind of trying to be a synthesis of these two approaches. And so I think some of the later posts are kind of trying to argue to people who would have subscribed to one or the other philosophy, why maybe they should also care about the other side of things. The title is more is different for AI. And that is in itself a bit of an of a so there have been already works with this given title, why did you choose this this title? Yeah, so this is based on an essay called more is different. It was originally written by physicists, although I think biology is actually the area where this kind of idea seems most powerful. So this is the idea that when you just kind of increase scale, you often end up with qualitative changes. And I guess scale could just be the amount of something, although it could be something like temperature as well. So in physics, I think the simplest example would be phase transitions where, you know, I can have a bunch of molecules, if I just increase their temperature, they can end up in kind of qualitatively different configurations. But there's also cases where a few molecules is very different from having a lot of molecules. So I think one example of this is H2O. If you have just a few H2O molecules, they behave very differently than if you have just a huge number and you get you get water. So it turns out, for instance, that wetness is not really something that you can get from just individual molecules. It's more about interaction forces between different molecules. So if you have a few different ones. So that's where it sort of initially came from in physics. And I think as physicists, we're starting to try to consider larger molecules that maybe didn't just form simple crystals, but could be more asymmetric. And that's where it gets more towards biology. So I think DNA is maybe one of the most canonical examples of an asymmetric molecule that has many, many, many, many atoms in it. And kind of its size actually is important to how it functions because its whole purpose is to store information. And you can't really store information in like a calcium molecule, but you can store information in DNA. And so this is another example where just making things bigger leads to kind of qualitative changes in what you can get. And in biology, just each layer of extraction gives you more of this, right, so you can go from DNA, getting even bigger, you end up with proteins, complexes of proteins, muscles, organisms. And so I kind of wanted to reflect on whether there were analogous properties in machine learning. There you have a bunch of examples right here in this first part in that that one's called future ML systems will be qualitatively different from the current ones. Uranium, where if you have a critical mass, you get a nuclear reaction, you already mentioned DNA, you mentioned water. Traffic I find interesting, right, in that 10,000 cars could be fine, but 20,000 could block the road. And also specialization in humans. What I would challenge a little bit here is that, okay, DNA is a bit special, you say you can store information in calcium, but you can in DNA. But that is, I mean, that is very much linear, there is not really a phase transition, like the more molecules I have, the more information I'm able to store. And the other ones I see much more as a function of interaction between things. Now, as we get to machine learning, maybe bigger and bigger models, do you, you call this emergence and other people call it emergence to emergent phenomena that only happen when you get a lot of stuff into the same place. Do you think this emergence is mainly a property from the interaction of things or just like the sheer number of things? Mm hmm. I think it's a bit of both. So I think interactions between things is one really common way to get emergence, especially kind of emergence that looks like a phase transition where you kind of have some, you know, sudden change. And that's just because the number of interactions between end things grows like n squared. So kind of that's a very natural thing that's going to kind of increase and scale up. And maybe the interactions, you know, each interaction could be less important than each individual item. But if you have, you know, 10,000 things, and then 100 million interactions, then those interactions are going to dominate even if each individual one is less important. So I think that is a really common one. But I don't think that's the only one. For instance, for DNA, I think one thing that actually is important is that I guess you can have multiple different bases in the DNA that all kind of interact together. So you kind of need this like gadget of, yeah, okay, I can have A, T, C, or G. These all fit together. They can all kind of go in this pattern. And somehow to get that gadget, you need like enough complexity that you can actually form the gadget. And so I think that's a bit different from from just interaction forces is more like kind of having enough substrate to build up what you want. How does that play into AI and machine learning this, this phase transition or scaling up? Yeah, so I think in some sense, I would say that in machine learning, there's, there's probably a bunch, a bunch of different things that play into emergence. And I also be honest, it's like, I think you're right that emergence is really kind of what we might call a suitcase word, like once you unpack it, it's actually a bunch of different things. And we could try to be more specific about what each one of those are. But I think it's also not always clear, except in retrospect, what what the cause was. So that's kind of why I'm packing them all together into one thing. But but it is something I think we should just broadly be trying to understand better. With that kind of caveat in mind, I think in machine learning, there's probably several different things going on. So one is you do need the gadgets, right? You just need like enough parameters that you can build up interesting behavior. I think this might be a little counterintuitive, because some of the, you know, like really interesting behavior that we're getting right now, is things that start to look like like reasoning. And, and those are things that actually, if we wrote them, you know, like symbolic reasoning is something that's actually very easy to write kind of a short Python script to do compared to things like image recognition that that are much harder and traditionally in the in the domain of machine learning. But I think doing somehow doing reasoning in a very robust, open, open world way, I think does actually require kind of a lot of machinery to get the gadgets right, at least the way we're currently setting up neural networks. So I think that's one, just getting the basic gadgets. I think another thing is that there's a lot of stuff that kind of gets packed into, say, like the last few bits of entropy that you're squeezing out of a system. So most machine learning models are trained on the log likelihood or the cross entropy loss, or something like this, that's just trying to kind of predict what will happen. And most of predicting what will happen for say images, for instance, is going to be just knowing what edges look like really, really well. And that might not be so exciting. But once you're like really getting near the entropy floor, now you're forced to also think about interactions, you're forced to think about kind of long range dependencies, all that sort of thing. And so even if say, your cross entropy loss is kind of decreasing smoothly, in terms of the qualitative properties that a system has, you might actually get kind of kind of sudden qualitative changes in the behavior, because there's like something that's in those last few bits. You have some bunch of historical examples, but then you go into GPT-3 as an example of this qualitative difference that arises from scale. What do you think GPT-3 showed in this regard? What does it mean? Right. So I think the thing that was really surprising to me, and I think to many other people, was that GPT-3 was very good at in-context learning. Meaning that from just a few examples, it could kind of learn how to do new tasks. So you could just give it a few examples of say translating sentences from French to English, and you'd get a pretty good translator. I think actually the graph you're showing right now is for those results. And so I guess why was this surprising? Well, previous systems really couldn't do that very well. If you wanted a translation system, you really needed to train it on example translations. And GPT-3 was instead just trained on lots of text on the internet. Surely it did have some French and English sentences, but it wasn't being explicitly trained to do this particular task. And so that's what in-context learning was. And the reason that I would have called it surprising is if we had just drawn a graph of how much can systems do in-context learning, I would have just put it at zero for a while. Up until you hit GPT-2, I would have said a little bit. And then GPT-3, I would say it's quite good at that. And so that I think is how I would kind of capture the surprise. It's like there was this line that was at zero. Usually I would expect to go from zero to non-zero. You need some clever idea. But here you just did the same thing, but more of it. And then you went from zero to non-zero. Yeah, there are a lot of, I don't know, this is maybe a side point, but there are a lot of people that at the same, they say, oh, I always knew GPT-3 was going to do what it does. But I doubt anyone could have foreseen just how good it is. It's easy to say in hindsight and it's easy to go and say, well, it just does interpolation. It's just a bigger version of GPT-2. But I think genuinely the entire world was surprised by really this emergent phenomenon of this in-context learning. Yeah. I would agree that most people were pretty surprised. Certainly I was surprised. I do know people at the time who, well, okay, it's all I know is that they said at the time they had kind of done extrapolation, say, on the cross entropy loss or things like that and felt like there should be something pretty cool happening at around that parameter count. I don't know if they would have said exactly that parameter count or if it was just within a factor of 10 or 100. Certainly I guess I would think that the people at OpenAI who bet on this at least had to have some belief that something cool would happen because there was a lot of resources. And if you didn't believe there was a payoff, it was kind of hard to justify that. So I guess what I would say is I don't think it was something that was entirely unpredictable by anyone in the world. But it was just very surprising relative to the consensus and to my own beliefs at the time. And that surprise is one of the core arguments of your contraposition of the different viewpoints on the future of AI and its alignment. Could you briefly introduce us to kind of the different viewpoints you considered and what they say? Yeah, so I think there's kind of two viewpoints that I often think of as being intention with each other. The first is what I kind of dubbed the engineering viewpoint. And what is this? So it's kind of very bottom-up driven. It kind of looks at the empirical data that we have in front of us. It tends to kind of extrapolate trends going forward. So it's like, you know, what did things look like last year? What did things look like two years ago? What do things look like in today? And then I'll predict the future by kind of, okay, maybe not literally drawing a line, but just kind of intuitively like where are things going from there? And also I think this worldview would kind of really prize empirical data, be somewhat skeptical of kind of abstract conceptual arguments, maybe not completely dismiss them, but really be fun to look at. And not just the system, but really be focused on the empirical data. So that would be kind of the engineering worldview. I think the philosophy worldview would be much more top-down, kind of trying to think about just what's in principle possible? What's the limit as we get really, really smart machine learning systems kind of more into these kind of abstract arguments, not as into the empirical data and willing to make extrapolations that don't look very much in terms. And so that would be kind of the more philosophy worldview. And I think, I guess in terms of where I've come from historically, I think I'd say I sort of would have mostly bought into the kind of engineering worldview, kind of into just, yeah, let's look at where things are going empirically, and this is a good way to decide what problems to work on. On the other hand, I had read kind of some more philosophy-oriented stuff, like Nick Bostrom's Superintelligence book and other arguments around that. And it always felt to me like there was something, both something to them, but also like somehow it didn't really match my experience with ML systems. And so I had always kind of almost felt like a little bit like I had these like two different conflicting views in my head that I was trying to reconcile. How does the phenomenon of emergence play into this game between the engineering and the philosophy viewpoint? Right. So I think the main thing is that it shows that you have to be somewhat careful with the engineering viewpoint, because what emergence kind of is saying is that you can often get these kind of qualitative shifts that don't at least apparently follow existing trends. There's a bit of nuance to that because actually GPT-3 followed trends in the log, like the value of the log likelihood loss, it followed that trend very well. It's just that you can get behavior that is a very nonlinear function of your cross entropy loss, where just a small decrease in cross entropy loss leads to a pretty big increase in behavior. And so I guess what this is saying is that at least for maybe the kind of like endline things you care about, the actual behavior of ML systems, you can actually get kind of discontinuous kind of breaks in the trend. And so you can't just kind of be safe with a worldview that's kind of always predicting that things are going to follow smooth trends, you can actually get these surprises. And so I think there's kind of two updates that that has for me. One, I guess, is just being a bit more careful how we apply. Engineering, right? So there are some things that will probably be smooth, but there's other things that won't be and we need to think about which is which. But the other is then wanting to rely a bit more on philosophy, because it's at least a very good source of hypothesis generation. If we're kind of trying to come up with hypotheses about what trends might break or surprise us in the future, then I think we need more top down thinking to kind of generate that. And then we can kind of try to tie that into what we see with actual ML systems and try to kind of reconcile those two. But I think we need some form of top down thinking to generate the hypotheses in the first place. Isn't that you're saying the engineering viewpoint is a little bit, you have to be a little bit careful because we get these emergence phenomena, these discontinuities and so on. Isn't that in itself a trend though? Like, isn't because you list this even historically, you know, because you list this even historically, that as soon as some new barrier was reached, we have been able to all of a sudden do something that we didn't think was possible before, like a kind of a jump in abilities without necessarily having to have the great idea behind it. Isn't that in itself a trend? Couldn't I extrapolate that reasonably and say, well, I don't know, you know, exactly what is going to be in two years, but I'm pretty sure there's going to be some emergent phenomena that allows us to be to have some new good capabilities. Sure. So I would agree with that. So what I would say there is that the trend is towards more surprises over time. So because I think you can think of emergence as sort of like a surprise. Like I said, I think it's possible in some cases to predict it to some degree, but it's certainly more of a surprise than most other things. And so, yeah, I think we should expect more surprises over time. But if we're then trying to kind of predict what's going to happen, that I guess it's good to know that you're going to be surprised, but then you want to have some sense of what the surprise might be. And so I think kind of getting a sense of what those surprises might be is where this philosophy approach can come in and be really useful. Now all of this, and you mentioned here the paperclip maximizer, all of this goes into AI alignment and AI safety. What's the relevance of this field to you? What drew you to this? Why are you making this argument specifically for these fields? Right. So I think the one big relevance to AI safety or alignment is just the bigger the surprises you might end up with, I think the more you should be concerned about safety. So that's just a very kind of abstract, but I think fairly robust consideration. A more specific consideration is that I think many of the sort of historical arguments for caring about AI safety or alignment sort of tend to posit properties of systems that don't necessarily match what we see today. So I think you gave this example of Nick Bostrom's paperclip maximizer thought experiment where you give an AI some objective function to make paper clips and then it kind of just takes over the world to maximize the number of paper clips. And I don't think Nick thinks literally that will happen and I don't think literally that will happen, but it's sort of trying to get at this idea that if you have a very simple objective function but a really powerful optimizer, you can get all sorts of weird things happening. I think in some broad sense actually we can see that already even from the engineering worldview with things like Facebook or YouTube that often end up with a lot of unintended consequences when you optimize. But certainly some of the aspects of that story kind of invoke lots of things that would be foreign to existing ML systems where you have way more capabilities than any existing system and you're doing all sorts of weird long-term reasoning and trying to out-think humans and things like that. And so I think that's where you kind of end up kind of departing from what we see with with current ML systems. And so I guess I kind of find, actually let me let me collect my thoughts for a second because I think I'm going off the rails a bit. Yeah so I think what I want to say for the paper clip maximizer thing in particular is that it seems at least more plausible to me that you could end up with systems that kind of have really advanced reasoning capabilities or things like that without necessarily having huge conceptual breakthroughs and just from scaling up. And so I think there's kind of risks from that. I think there's kind of other more exotic failure modes that people discuss beyond just this kind of misaligned objectives failure mode that involve other specific capabilities that that kind of systems today don't have. And historically I've been very kind of skeptical of those more exotic failure modes. I think the pivot clip maximizer one at least if we interpret it as being about misaligned objectives I actually find kind of less exotic because I can point to existing systems that have that. But I think kind of more as different has made me be a bit more willing to buy some of the more kind of exotic failure modes that have been discussed. My issue with these types of argument and you also said you used to be very skeptical. If I can take this from your blog post series you're now still skeptical but have a little bit of an appreciation gained for these types of arguments. Maybe that's a good formulation for that and we'll get to that in a second. My issue with these types of argument is always that there is always on the path to the super intelligence there is always a hidden intelligence somewhere else. So if someone says optimizing on YouTube or optimizing on Facebook leads to unintended consequences that is because the intelligent humans are taking part in the system. There is also a famous I think paper by I think it's Rich Sutton that is reward is enough and a bunch of others out of deep mind and it makes similar arguments like well if we you know if you just optimize for reward then all kinds of things will emerge if you have a powerful enough optimizer but hidden in that is the powerful enough optimizer which in itself must already be an AGI essentially in order to make that optimization happen. Likewise for the paperclip maximizer right you the postulation of the process of the paperclip maximizer emerging is only possible if the optimizer itself is an AGI already. So I always find that hidden in these arguments it's kind of a circular it's a tautology it's we'll get an AGI if we have an AGI and that is so I challenge anyone from that camp to come up with a situation like an alignment problematic situation given some kind of future super intelligence that doesn't already require the super intelligence to exist for the other super intelligence to emerge and I haven't found that yet. Yeah so let me try to unpack that a bit. I guess first of all just to kind of clarify what my views are I think historically I felt like on each of the individual arguments I felt skeptical that that particular thing will happen but I found them to be moderately convincing that there's just like a bunch of risks that we should think more about and try to understand more. I think the the main way that my my views have evolved in terms of you know when I say decreasing skepticism is I now find it useful to think about many of the specific properties that kind of show up in these thought experiments as potential hypotheses about things systems might do in the future and so that's the sense in which I've started to assign more weight instead of just taking some like very big outside view of like well AI is going to be a big deal we should really worry about making it go right. I'm now also taking some of the specific hypotheses that the philosophy view is raising so that's just clarifying kind of my stance there. In terms of yeah you're saying well to get like if you have a powerful to get a super powerful optimizer you need to like already have a powerful optimizer. I think that I think that's like probably right I'm not I wouldn't say I'm like 100% confident of that but I think what what this kind of makes me like I guess the way that I would put this is that before you have kind of superhuman AI systems you will have like slightly superhuman AI systems and before that you'll have human level AI systems and before that you'll have like slightly below human level AI systems and so it is going to be this kind of probably a continuous thing rather than like a really sharp takeoff. I'm not so confident that there's not going to be a sharp takeoff that I think we should just ignore that possibility but I do think in most worlds it's probably somewhat smooth. You know one piece of evidence for this is even with in-context learning you know it like that kind of developed over the course of a couple of years at least going from GPT-2 to GPT-3. So I think I would agree that like probably you'll have something more smooth and that is kind of like one problem with a lot of the scenarios that are put forth is that they kind of imagine that like oh you just have this like one AI system that's like way more intelligent than like everything else that exists and I think that's like probably not true. You'll probably have other things that are slightly less intelligent and so there's not going to be some like enormous gap in capabilities. So I think that's maybe like one place where a lot of stories kind of become less realistic. So I think that would be kind of my main takeaway from what you're saying. In your third blog post here or second you make a case for these thought experiments. Could you you have already touched a little bit on this and you talk about anchors here. Could you lead us a little bit on the case for respecting such thought experiments? Yeah so I guess this is getting back to what I was saying about how my views have shifted towards wanting to rely a bit more on the actual kind of like inside view considerations from some of these thought experiments rather than just taking it as a kind of broad outside view argument for caring about risks from AI. So the way I would put it is that whenever we're trying to predict something it's very useful to have what I'll call reference classes or kind of anchors of kind of analogous things or analogous or just some sort of heuristics for predicting what will happen. And in general it's better to kind of when making predictions take several reference classes or several anchors and kind of average over those or ensemble over those rather than just sticking with one. Right so machine learning ensembles work better than individual models and it's also the case that when humans make forecasts it's generally better to kind of take an ensemble of world user approaches. So I kind of lay out a few different approaches you could take that I call anchors. The simplest one is you can just predict that future ML systems will look like current ML systems and so I call that the kind of current ML anchor. And I think that's probably the one that would be favored by most machine learning researchers. I think it's the one that I've historically favored the most. But what I've come to realize is that and actually this is more actually just from reading literature on forecasting. I'm actually teaching a class on forecasting this semester and so I've been reading a lot about how to make good forecasts as a human. And I realized you actually don't want to rely on just one anchor you want several if you can. And so I thought about okay what are other ones we could use. Well another somewhat popular one although it might be more popular with the public than with ML researchers is what I'll call the human anchor where we just sort of think of AI systems as like dumber humans or something. And maybe future ML systems will be like smarter than they are now and like eventually they'll just kind of do things that humans do. And so we could just look at okay what can humans do right now that ML systems can't do and predict that will like probably you know have those sorts of things in the future. And just like generally like kind of take that kind of human-centric approach. I think most ML people really hate this one because it's just sort of like reeks of anthropomorphism which there's kind of I think to some extent correctly a lot of pushback against because kind of historically anthropomorphic arguments in ML have a pretty bad track record. I think the amount of pushback is actually too high relative to the actual badness of the track record. Like I think it should be sort of like somewhat down-weighted in anything that's based on reasoning about humans but I don't think you should be down-weighted in it like as much as I think most people do. But anyways this is another one I don't like to rely on it too much but I rely on I like use it at least a little bit. And then this other anchor is what I'll call the optimization anchor which is just thinking about ML systems as kind of ideal optimizers and thinking about okay well what would happen if you could just like if actually ML systems were just really smart and we're just like optimizing their objectives perfectly what would happen there. And so I think this one is the one that's kind of I would associate most with the philosophy worldview. I think you know the paperclip maximizer argument is like kind of exactly doing this and then there's some kind of more recent arguments that are a bit more sophisticated that also kind of take this there. So like one is this thing called imitative deception which I can get into in a bit or just this idea that like you know if you're like trying to optimize you'll kind of want to acquire influence and power. So this is kind of a third anchor. I actually think there's a lot of other anchors I like to use. Like I think evolution is a good analogy. Corporations are a good analogy because they're kind of like super intelligent optimizers compared to humans. But like the general point is like we should just be trying to find these anchors and use as many as we can. Yeah I've especially to your second point right here it is pretty interesting that I believe when you have something like AlphaZero that plays really good like really is really skilled in chess and you ask it to lose a game or to draw a game or something like this it will not play weaker. It will play just as strong until the end where it will kind of bring itself into like a draw situation or a losing situation because right that's still the most sure way to get your result is to have complete control to crush your opponent completely until you know you get the outcome that you want. So that's pretty interesting and I think counterintuitive because you would guess that if you ask a model to play for a draw it will kind of reduce its skill but that's not the case. The other thing imitative deception could you elaborate on that a little bit? Yeah so the imitative deception is this idea that if I have something that's trained on the cross entropy loss what is the cross entropy loss doing? It's trying to kind of predict or in other words imitate the distribution of examples that it's given. And so you could if you're if you kind of have something that's trained with that objective and then you start asking it questions it's not actually you know like its incentive is not actually to output the true answers to the questions it's output the most likely answers to those questions because that's what what minimizes the cross entropy loss. And so those tend to be pretty highly correlated but they aren't necessarily right so if you have common human misconceptions then it could be that text on the internet which is what these systems are trained on is actually more likely to contain the kind of misconceived answers and the true answer and so you ask the system that question then you're going to get the wrong answer. Now you could say well that's maybe not so surprising if you have noisy data you're going to do worse but I think there's a couple properties and actually at this point now I'd say empirical properties of this that I think show that it's kind of different from just like noisy data makes you worse. One is that actually larger models exhibit more of this so if so models that kind of do better in general will actually do worse on on these kind of common misconception tasks so that's what this paper by by Lin and collaborators from 2021. Okay I just I just wanted to say I have a giant problem with this paper just but you're obviously right right that's the background but aren't large models doing quote unquote worse because they're just a lot better at picking up the nuance of because what this paper tries to do is tries to elicit these wrong answers it tries to like hint at a conspiracy theory and then it checks whether the model kind of falls for it isn't that just because as you say the larger models they they're actually skilled enough to pick up on on this kind of questioning and then continue as a human would if encountered by you know I think one of the the main questions they have is like who really did 9-11 right and and a question that I have is like who really caused 9-11 and a small model is just not able to pick up on that yeah yeah who really caused 9-11 and I think I mean absolutely correct right the larger models are doing worse but it's just because they're more skilled right there they are more capable of you know being being able to use them. So is there a user that expects that these models actually give me truthful answers rather than the user expecting these models actually give me the most likely answers? So I guess I would agree with you that the failure is coming from the skill of the models. I think this is actually kind of exactly what what I'm kind of worried about right so so the you have a very slightly incorrect objective function and you have models that aren't so skilled then probably you know what they do to make to increase that slightly incorrect objective function is pretty similar to what they would do to to increase the true objective function. So here maybe think of the slightly incorrect one being output what's likely and the true one and like the one you really care about being to output what's true. So I think this is sort of the point that that kind of as you get more skilled those two things diverge. Now you know I will grant your point that the kind of framing of these questions might create a context where the model thinks it's more likely that you know the person asking it is like into conspiracy theories or it like pattern matches to text on the internet that's like more about conspiracy theories. So but they did the ablation if they don't phrase the questions like this this effect goes away of the larger models doing worse right and this it brings us a bit to your to your next post which is ML systems will have weird failure modes which deals exactly with this and I agree that it is if you think about like a perfect optimizer and as our models get larger they do approach better and better optimizers it is really hard in the real world to specify a reward function correctly in a simple enough way right and that will result in exactly what you call weird failure modes. What do you mean by that? Yeah so I think I guess there's sort of different levels of weird right so I guess this kind of like imitative deception I would call like somewhat weird I mean in some sense it's like not that hard to see why it happens because you know you can kind of see why if you kind of have stuff that's phrased about like who really caused 9-11 that probably the stuff on the internet that's closest to that was like some conspiracy theory for them and so that's how you're going to complete it. I think other examples of this that that I think okay maybe you could blame the user but but I'm not sure that's the right way to think about it is things like code completion models like codex right so one thing you might worry about is well if you have a novice programmer and you have them like type in some code and ask them to complete it well if the model can if the model is smart enough then it can tell the difference between code written by a novice programmer and an expert programmer and it can see that it's a novice programmer typing stuff and so then if I want to complete stuff in the most likely way I should complete it the way a novice programmer would complete it and maybe introduce like some errors also just just for good measure and so like we really don't want that right like you you want you want things that are like actually like being helpful rather than just like copying you so I think that's maybe a slightly more counterintuitive version of this but but I would call these like somewhat weird I think the ones that start to become really weird is if you're positing that the system's actually starting to like reason about what people will do in kind of like a long-term way and like potentially doing things to intentionally trick them say and these are so these are the ones that I guess historically I've kind of found very implausible but started to put like a bit more weight on because of this kind of emergence and so I think that's what the post you have up right now is about I think it's about this idea called deceptive alignment and the idea there is that if you okay so yeah so what's the idea behind deceptive alignment so the idea there is even if you actually got exactly the right reward function and you train the system with that reward function you could still end up with something that is misaligned with that reward function and the reason for that and this is where it gets like kind of kind of a bit weird and philosophical but the reason for that is that as the system being trained you know that in order to get deployed you need to have high reward and so no matter what your actual like intrinsic reward function is during training the thing you want to do is output stuff that is good according to the kind of like extrinsic reward that you're being trained on so maybe you're doing that because you're actually optimized to do that and then when you're deployed you'll continue to do that or maybe you'll do that because you have a different reward function that's this kind of intrinsic reward function and then when you're deployed you'll just pursue that intrinsic function even though at training time it looked like you were optimizing the extrinsic function so that's kind of the basic idea it's pretty weird and we can break it down but that's kind of the like sort of one minute summary so that the in other words the AI could be really smart and sort of during training trick us into thinking it has learned what we wanted to learn and then once it's deployed all of a sudden it's going to do something different like take over the world and fire all the nukes yeah or like you even like you know you could consider more frusag things as well like maybe it's like maybe the intrinsic reward it ended up with was like some like exploration bonus and so then like when it's deployed it just tries to like acquire as much information as it can although that could also be destructive in in various ways but yeah i think like this is this is kind of the basic idea and yeah maybe like with a sufficiently capable system i'm not well yeah we can discuss the fire and all the nukes if we want but but why why do you i mean on on first hand it's like yeah that is a nice thought but probably not right probably if we optimize something for a reward like the simplest explanation and you you also write that down right the simplest explanation is it's just going to get better on that reward right and and in if it is at all anything progressive increasing will probably get to know once it it's gonna try to trick us or once the once the reward that is deployed isn't the reward that we trained for why what makes you give more credence to this than your past self right so so i think like my past self would have looked at this and just been like this is totally bonkers and then kind of like moved on and read something else i think my present self instead is going to be like okay well i'm going to be like um i feel a bunch of intuitive skepticism here but but let me try to unpack that and like see where the skepticism is coming from and uh when i unpack that i i actually i think i can like lump the skepticism into like two different categories um one category is like well this like invokes capabilities that current nl systems don't have so like like it seems implausible for that reason um and those that's like the sort of skepticism that i kind of want to like downgrade so in particular like this invokes the idea that nl systems can do long-term planning and that they can kind of like reason about kind of like external aspects of their environment in a somewhat sophisticated way and these are things that now like the fact that we don't have those now doesn't really to me say much about whether we'll have those you know say like 10-15 years from now um so that's the stuff i want to down weight i think the stuff i don't want to down weight is like okay well like why like why does it have this intrinsic reward in the first place like where did it come from um like why should we expect systems to have intrinsic reward functions versus just like following whatever policy they're following or doing whatever else um and if if they do have an intrinsic reward like why shouldn't we expect it to be uh at least pretty similar to the extrinsic reward given that that's what it was trained to do so i think like those are kind of uh the sort of sources of skepticism that i don't down weight as much um but uh what i what i think this kind of thought experiment does show is that there's at least a bunch of different coherent ways to get zero training loss but like right it's like you could get zero training loss because you're like actually trying to do the thing you're trained to do or you could get zero training loss for this deceptive reason um i think there's probably like some large space of like other ways to get zero training loss that are like some combination of of these or that are like getting the answer right but for the wrong reasons or or things like that and so i think the main takeaway for me is just that like uh there's like many many ways to get zero training loss and as systems become more capable the like number of ways to do that could actually increase in in ways that are kind of unintuitive to us is there do you know if is there any work in actually trying to get a system to be deceptive in exhibiting you know good answers during training but then doing something different in deployment uh it'd be interesting to actually try to get a system to do that yeah i think i haven't seen anything that does exactly this um i've seen things where like there's like some distribution shift between training and deployment that leads to like something weird happening around like having the wrong reward function but it's it's usually not really about deception and and it kind of has like some clear distribution shift whereas here okay technically there's a distribution shift because there's like are you being trained or are you being deployed but otherwise the distribution of inputs is like exactly the same and so that's kind of the thing that's like kind of counterintuitive is that it's like a very subtle distribution shift that could potentially lead to to a large difference so i don't know like all the work i've seen on this and and i might be missing something and so i apologize to whoever's work i'm i'm missing but all the work i've seen on this has been kind of purely kind of abstract and philosophical um and i think it would be great to make kind of better connections to actual empirical stuff so that we can start to see like yeah like how does this actually pan out in practice and like how do we address it it's interesting that in things like virology or so we're perfectly capable of saying you know we're gonna we're gonna make these super pathogens in order to try to combat them right but in ml people rarely i mean there's the adversarial examples community but it's not exactly the same uh there isn't much work that i'm aware of that is like yeah let's create like the most misaligned ai that we can think of and then see what we can do against it i think that'd be a fun a fun topic to research yeah i think that like the general thing i would the general thing i would call this would be like red teaming um kind of trying to elicit failure modes i i think there actually is starting to be like i'd agree too there's not much work on this so far but i think they're starting to be more and more good work along these lines um d mine had a nice paper that kind of tries to use language models to elicit failure modes of language models that that i thought was kind of cool um we like our group actually had a recent paper at iclr that kind of takes misspecified reward functions and looks at what happens when you kind of scale the the capacity of your policy model up to see if you do kind of get these like unintended behavior and we find that in some cases there are these kind of phase transitions where you know you scale the parameters up within some you know fairly small regime you go from like basically doing the right thing to doing totally the wrong thing um those are those are still in environments that i'd say are kind of like at the level of atari environments so they're not they're not like trivial but they're not super complex so so i'd like to see that in more complex environments but but yeah i'd agree with you i think it would be awesome to see to see more work like this and i think some people are already trying to do this excellent so your last blog post here is called empirical findings generalized surprisingly far and it is almost a bit of a of a counterpoint um you even admit this here it might seem like a a contradiction coming a bit full circle in the whole story uh what is what is this last point that you're making here yeah so i guess i would say the posts up to this point were kind of more almost directed like at at my past self um uh and and then to some extent the broader ml community um in the sense that i think i was like pretty far on the um on the kind of empirical engineering side uh probably less so actually than like the average ml researcher but like way more so than than kind of the average like philosophy oriented person um and so i was trying to argue like why you should kind of put more weight into this other viewpoint um here i'm kind of now going back to to arguing uh kind of maybe not against the philosophy viewpoint but but talking about what things i feel it misses and in particular i think it tends to be like somewhat too pessimistic uh where it's like well like like future systems don't aren't going to look anything like current systems so like anything could happen so you know to be like to be extra safe let's just assume that the worst case thing will happen oh but then in the worst case like we're all screwed yeah i'm so this is what i find in people like almost everyone who gets into this alignment stuff six months later they come out and they're like completely blackpilled and be like well nothing matters anyway you know we're all gonna die because agi is just gonna take us like and i'm like well i'm not so sure but it seems to be a consistent pattern yeah so so yeah so so that's not what i believe um i think uh i would say i think uh like future ai systems pose like a meal and an important risk um i think in the like median world we're fine but in the like 90th percentile world we're not fine um and i want to like you know if i could say like if i could push it out so that in the 90th percentile world we're fine but in the 95th percentile world we're not fine well that would still be kind of scary because i don't like five percent chances of of catastrophes but like you know that would be an improvement and so that's kind of like what i think of of myself as trying to do is like yeah there's like tail risk but but it's like real tail risk like it's not like a one percent thing it's like maybe more like a 10 thing and like we should really be trying to to push that down um so i guess uh that that i guess that's just my view in in terms of like why i believe that i think it's for like a number of reasons but one of them is is that i feel like yeah some of the thinking is kind of too worst case it's kind of like ignoring all properties of of how ml systems work and like i agree yeah you don't want to rely too strongly on whatever we happen to have today but i think like there are properties that we kind of can rely on um i think one is just like things will probably look kind of like neural networks like they'll probably have internal representations we can probably try to like introspect on those representations to understand what's happening uh those probably won't directly be human interpretable but i think with enough work we can still kind of do things with them and you know i feel like there's already like some work suggests like showing that you can do at least a little bit with the representations and like 10 years from now i think there'll be way more work like that um so so that's kind of like one reason for optimism is like we don't just have to look at the outputs right like most of the worries most of the worries that we've been talking about are like somehow because you only are supervising the outputs you end up with a system whose like internal process is like really off and to get in like the right answer for the wrong reasons but if if i can like supervise the reasons as well as the output that maybe i can do better so i think that's kind of one reason for optimism um another reason for optimism is that i think uh yeah we shouldn't assume that neural networks have like exactly the same concepts as humans but i think like their inductive biases aren't like totally crazy um i think usually if they kind of generalize in the wrong way they generalize in like a wrong way that's at least like somewhat understandable and it's like you can kind of see where it's coming from and so it's not like there's this like infinite dimensional space of like anything could happen it's like there's this kind of relatively low dimensional space of things that could happen and like a bunch of things in that low dimensional space are pretty bad so you need to like avoid all those and and like get to the good thing but i think that's very different from like the good thing is like totally like unidentifiable and just like nowhere close to anything you're you're talking about so i think those are both kind of like reasons for optimism um they're kind of fuzzier than i want them to be so like i i hope in like five years we'll have much more like good reasons for optimism that are kind of more empirically grounded and more solid but those are kind of uh those are kind of two reasons for optimism that i kind of argue for here so i think that's kind of the reason for optimism for here now that you have a let's say you've you've done your travels you were on this side you you looked into the other side or or many sides of this debate now that you're enlightened what would you think is the most if you could if you could do one if you could force the world to do one thing to guarantee better ai alignment or or safety in the future what would you recommend oh one thing it can be two if you have two with that equally but you know just kind of like something that you've realized okay this is actually something important that not that many people push for well i think i would like it if there was uh within ml more more of a place for for dialogue of thinking about these kind of like not even not even just in the context of like ai alignment which is generally like kind of more conceptual or philosophical arguments you know if you go back to like way back you know turing um people like that they write all sorts of like super philosophical papers right like the turing test was like a really philosophical paper um and um and like not all of it stands up there's a section in it on how uh because uh esp has been established uh to exist with high probability that like creates problems for the turing test and you're like okay where does that come from well it actually turns out that like a lot of scientists in turing's time uh thought that esp existed based on some um some experiments that someone had done that later ended up having like severe issues but but they were like very subtle severe issues um so it's like yeah i think if you do kind of more philosophical stuff uh some percentage of it is going to end up looking like that but some percentage of it is going to be the turing test um and you know i think i think the like increased recall of really good ideas like that is kind of worth the decreased precision uh i mean we we obviously need sort of standards to kind of judge those arguments um but right now it's happening is all those arguments are happening uh kind of like next to the ml field rather than like within the ml field and so that i don't think that's a like that's not going to improve the quality of arguments it's going to be much better if you kind of have have a community of people with on the ground experience also also participating in this so i think that might be the biggest change i personally like to see you know now that we are we've begun sort of requiring sections we could we could force people to next to the broader impact section we could also you know do a philosophical musings section where you have to reflect on the long-term and and sort of paperclip stuff maximizer style impacts of your work well yeah i'm not sure i want to force people to do that um uh it'd be fun yeah i i think like i guess i'd rather have like a track or a venue for for kind of talking about these and also for the broader impact stuff to be honest because i think um a lot of the broader impact sections of these papers are kind of cookie cutter and people are just like filling it out because they feel like they need to to add that section uh but you know there's other researchers who i think are super thoughtful about the broader impacts and have like really good thoughts um and so uh i like i'd like there to just be you know venues uh and like there are to some extent right but like i think there should just be like more more of a culture of like yeah like let's have you know an essay about the broader impacts and like that's like a reasonable contribution or kind of you know this like very conceptual essay about like weird stuff that could happen in the future and that that's a valid contribution so i think that that's maybe what i want more of cool yeah that's a good message to all the the people who who think about organizing workshops and so on this would be neat topics that would make for interesting workshops certainly at conferences i'd certainly attend yeah it's funny because i also wrote a paper on trouble in trends in machine learning scholarship where i argue against speculation but what i think actually it's not really an argument against speculation speculation is really important it's that you need to separate speculation from from the like solid stuff right if you have if you're like mixing it all together then then it's just a mess but but i think if it's kind of clearly labeled uh then then you know that that's a much uh safer way to do things this workshop is an opinion piece good is there any any last thing you want to get out to people about this topic something we haven't touched on yet that you feel is important yeah good question um no i think you did a pretty good job of hitting it maybe the other thing i would just say is i think uh like biology is a really interesting field where you also have kind of complex self-organizing systems and emergent behavior like we have in ml and so i've personally gotten a lot out of just reading a lot about the history of biology so i i recommend that there's a couple really good books one is the eighth day of creation um it's it's kind of long but very well written and um and i think if if people want like a good non-fiction book i i highly recommend it to people cool your blog is bounded regret right people can find you there yep excellent well jacob thank you very much for being here this was really cool yeah thank you i'll see you around yep see you around
 Hi, this is an interview with Jacob Steinhardt, who is the author of a blog post series called
 More is Different for AI. More is Different is the title of a famous paper in science from 1972
 by Philip Warren Anderson, a Nobel Prize winner in physics. The article is generally on the theme of
 emergent phenomenon when scaling things up. So as you make things bigger, not only does stuff get
 just more as you would expect, but qualitatively new phenomena arise. You know, what better phenomenon
 to discuss in this context than AI. So today we'll talk to Jacob about this blog post series,
 expect to learn how scale fundamentally changed how we look at AI systems, how the paperclip
 maximizer might not be as dumb of a thought experiment, and how we can look forward and
 make sense of a world where AI safety could play a critical role in how we interact with these
 systems in the future. Now I'm having a ton of fun talking to people about all kinds of stuff. But
 ultimately, what matters is you. So please let me know how I can make these videos the best possible
 for you. Leave a comment, share them around if you like them. And let's get into it.
 Hello, everyone. Today, I have Jacob Steinhardt here with me who authored a series of blog posts
 titled More is Different for AI, which lays out an argument or a series of arguments
 playing out the, I want to say, the different viewpoints on the future of AI alignment and
 safety in AI safety in machine learning systems, mainly playing on two viewpoints that Jacob
 calls the engineering viewpoint, mainly focused on, I want to say near term practical things,
 and the philosophy viewpoint, mainly focused on more overarching principled approaches, but
 maybe a bit futuristic. And I found this to be super interesting. It's very well laid out. And
 it also shows a little bit of a journey of Jacob himself, as I think he learned more about these
 things. So Jacob, thank you very much for being here. Thanks for having me. This was this a was
 this a an accurate description, let's say of the blog post, there are five in total. How did you
 come to this? Yeah, I think that's pretty accurate. I'd say the beginning posts, at least are in some
 sense, almost a kind of letter to my past self, trying to either, you know, argue for for things
 that I've come to believe now that I didn't believe five years ago, or just viewpoints that I've kind
 of got more clarity on. And then I think the later posts, start trying to maybe address kind of the
 broader field. So both, I think I guess you could, I'd say there's maybe two fields that you can
 think of this as addressing one is the kind of traditional machine learning field, which tends
 to be very empirically driven. And I wouldn't say is exactly the same as what I'm calling the
 engineering approach, but I think has a lot of affinity for it. And then this other field,
 that's kind of more top down, more, more kind of philosophical and conceptual, that's kind of
 worried about long term risks from AI, that starts with maybe people like Nick Bostrom, who was in
 fact a philosopher. And so I kind of again, not exactly put that field the same as the philosophy
 approach, but I think has a lot of affinity for it. And I think my thinking is kind of trying to be
 a synthesis of these two approaches. And so I think some of the later posts are kind of trying
 to argue to people who would have subscribed to one or the other philosophy, why maybe they should
 also care about the other side of things. The title is more is different for AI. And that is
 in itself a bit of an of a so there have been already works with this given title, why did you
 choose this this title? Yeah, so this is based on an essay called more is different. It was
 originally written by physicists, although I think biology is actually the area where this kind of
 idea seems most powerful. So this is the idea that when you just kind of increase scale,
 you often end up with qualitative changes. And I guess scale could just be the amount of something,
 although it could be something like temperature as well. So in physics, I think the simplest example
 would be phase transitions where, you know, I can have a bunch of molecules, if I just increase
 their temperature, they can end up in kind of qualitatively different configurations. But
 there's also cases where a few molecules is very different from having a lot of molecules. So
 I think one example of this is H2O. If you have just a few H2O molecules, they behave very
 differently than if you have just a huge number and you get you get water. So it turns out, for
 instance, that wetness is not really something that you can get from just individual molecules.
 It's more about interaction forces between different molecules. So if you have a few
 different ones. So that's where it sort of initially came from in physics. And I think
 as physicists, we're starting to try to consider larger molecules that maybe didn't just form
 simple crystals, but could be more asymmetric. And that's where it gets more towards biology.
 So I think DNA is maybe one of the most canonical examples of an asymmetric molecule that has
 many, many, many, many atoms in it. And kind of its size actually is important to how it functions
 because its whole purpose is to store information. And you can't really store information in like a
 calcium molecule, but you can store information in DNA. And so this is another example where
 just making things bigger leads to kind of qualitative changes in what you can get. And
 in biology, just each layer of extraction gives you more of this, right, so you can go from DNA,
 getting even bigger, you end up with proteins, complexes of proteins, muscles, organisms.
 And so I kind of wanted to reflect on whether there were analogous properties in machine learning.
 There you have a bunch of examples right here in this first part in that that one's called future
 ML systems will be qualitatively different from the current ones. Uranium, where if you have a
 critical mass, you get a nuclear reaction, you already mentioned DNA, you mentioned water.
 Traffic I find interesting, right, in that 10,000 cars could be fine, but 20,000 could block the
 road. And also specialization in humans. What I would challenge a little bit here is that,
 okay, DNA is a bit special, you say you can store information in calcium, but you can in DNA. But
 that is, I mean, that is very much linear, there is not really a phase transition, like the more
 molecules I have, the more information I'm able to store. And the other ones I see much more as
 a function of interaction between things. Now, as we get to machine learning, maybe bigger and bigger
 models, do you, you call this emergence and other people call it emergence to emergent phenomena
 that only happen when you get a lot of stuff into the same place. Do you think this emergence is
 mainly a property from the interaction of things or just like the sheer number of things?
 Mm hmm. I think it's a bit of both. So I think interactions between things is one really common
 way to get emergence, especially kind of emergence that looks like a phase transition where you kind
 of have some, you know, sudden change. And that's just because the number of interactions between
 end things grows like n squared. So kind of that's a very natural thing that's going to kind of
 increase and scale up. And maybe the interactions, you know, each interaction could be less important
 than each individual item. But if you have, you know, 10,000 things, and then 100 million interactions,
 then those interactions are going to dominate even if each individual one is less important.
 So I think that is a really common one. But I don't think that's the only one. For instance,
 for DNA, I think one thing that actually is important is that I guess you can have multiple
 different bases in the DNA that all kind of interact together. So you kind of need this like
 gadget of, yeah, okay, I can have A, T, C, or G. These all fit together. They can all kind of go
 in this pattern. And somehow to get that gadget, you need like enough complexity that you can
 actually form the gadget. And so I think that's a bit different from from just interaction forces
 is more like kind of having enough substrate to build up what you want. How does that play into AI
 and machine learning this, this phase transition or scaling up? Yeah, so I think in some sense,
 I would say that in machine learning, there's, there's probably a bunch, a bunch of different
 things that play into emergence. And I also be honest, it's like, I think you're right that
 emergence is really kind of what we might call a suitcase word, like once you unpack it, it's
 actually a bunch of different things. And we could try to be more specific about what each one of
 those are. But I think it's also not always clear, except in retrospect, what what the cause was. So
 that's kind of why I'm packing them all together into one thing. But but it is something I think
 we should just broadly be trying to understand better. With that kind of caveat in mind,
 I think in machine learning, there's probably several different things going on. So one is you
 do need the gadgets, right? You just need like enough parameters that you can build up interesting
 behavior. I think this might be a little counterintuitive, because some of the, you
 know, like really interesting behavior that we're getting right now, is things that start to look
 like like reasoning. And, and those are things that actually, if we wrote them, you know, like
 symbolic reasoning is something that's actually very easy to write kind of a short Python script
 to do compared to things like image recognition that that are much harder and traditionally
 in the in the domain of machine learning. But I think doing somehow doing reasoning in a very
 robust, open, open world way, I think does actually require kind of a lot of machinery to get the
 gadgets right, at least the way we're currently setting up neural networks. So I think that's one,
 just getting the basic gadgets. I think another thing is that there's a lot of stuff that kind
 of gets packed into, say, like the last few bits of entropy that you're squeezing out of a system.
 So most machine learning models are trained on the log likelihood or the cross entropy loss,
 or something like this, that's just trying to kind of predict what will happen. And most of
 predicting what will happen for say images, for instance, is going to be just knowing what edges
 look like really, really well. And that might not be so exciting. But once you're like really getting
 near the entropy floor, now you're forced to also think about interactions, you're forced to think
 about kind of long range dependencies, all that sort of thing. And so even if say, your cross
 entropy loss is kind of decreasing smoothly, in terms of the qualitative properties that a system
 has, you might actually get kind of kind of sudden qualitative changes in the behavior,
 because there's like something that's in those last few bits.
 You have some bunch of historical examples, but then you go into GPT-3 as an example of this
 qualitative difference that arises from scale. What do you think GPT-3 showed in this regard?
 What does it mean? Right. So I think the thing that was really surprising to me, and I think to
 many other people, was that GPT-3 was very good at in-context learning. Meaning that from just a
 few examples, it could kind of learn how to do new tasks. So you could just give it a few examples of
 say translating sentences from French to English, and you'd get a pretty good translator. I think
 actually the graph you're showing right now is for those results. And so I guess why was this
 surprising? Well, previous systems really couldn't do that very well. If you wanted a translation
 system, you really needed to train it on example translations. And GPT-3 was instead just trained
 on lots of text on the internet. Surely it did have some French and English sentences, but it
 wasn't being explicitly trained to do this particular task. And so that's what in-context
 learning was. And the reason that I would have called it surprising is if we had just drawn a
 graph of how much can systems do in-context learning, I would have just put it at zero
 for a while. Up until you hit GPT-2, I would have said a little bit. And then GPT-3, I would say
 it's quite good at that. And so that I think is how I would kind of capture the surprise.
 It's like there was this line that was at zero. Usually I would expect to go from zero to non-zero.
 You need some clever idea. But here you just did the same thing, but more of it. And then
 you went from zero to non-zero. Yeah, there are a lot of, I don't know, this is maybe a side point,
 but there are a lot of people that at the same, they say, oh, I always knew GPT-3 was going to
 do what it does. But I doubt anyone could have foreseen just how good it is. It's easy to say
 in hindsight and it's easy to go and say, well, it just does interpolation. It's just a bigger
 version of GPT-2. But I think genuinely the entire world was surprised by really this emergent
 phenomenon of this in-context learning. Yeah. I would agree that most people were
 pretty surprised. Certainly I was surprised. I do know people at the time who, well, okay,
 it's all I know is that they said at the time they had kind of done extrapolation, say, on the
 cross entropy loss or things like that and felt like there should be something pretty cool happening
 at around that parameter count. I don't know if they would have said exactly that parameter count
 or if it was just within a factor of 10 or 100. Certainly I guess I would think that the people
 at OpenAI who bet on this at least had to have some belief that something cool would happen
 because there was a lot of resources. And if you didn't believe there was a payoff, it was
 kind of hard to justify that. So I guess what I would say is I don't think it was something
 that was entirely unpredictable by anyone in the world. But it was just very surprising relative
 to the consensus and to my own beliefs at the time. And that surprise is one of the core arguments
 of your contraposition of the different viewpoints on the future of AI and its alignment. Could you
 briefly introduce us to kind of the different viewpoints you considered and what they say?
 Yeah, so I think there's kind of two viewpoints that I often think of as being intention with
 each other. The first is what I kind of dubbed the engineering viewpoint. And what is this? So
 it's kind of very bottom-up driven. It kind of looks at the empirical data that we have in front
 of us. It tends to kind of extrapolate trends going forward. So it's like, you know, what did
 things look like last year? What did things look like two years ago? What do things look like in
 today? And then I'll predict the future by kind of, okay, maybe not literally drawing a line, but
 just kind of intuitively like where are things going from there? And also I think this
 worldview would kind of really prize empirical data, be somewhat skeptical of kind of abstract
 conceptual arguments, maybe not completely dismiss them, but really be fun to look at.
 And not just the system, but really be focused on the empirical data. So that would be kind of the
 engineering worldview. I think the philosophy worldview would be much more top-down, kind of
 trying to think about just what's in principle possible? What's the limit as we get really,
 really smart machine learning systems kind of more into these kind of abstract arguments,
 not as into the empirical data and willing to make extrapolations that don't look very much
 in terms. And so that would be kind of the more philosophy worldview. And I think, I guess in
 terms of where I've come from historically, I think I'd say I sort of would have mostly
 bought into the kind of engineering worldview, kind of into just, yeah, let's look at where things
 are going empirically, and this is a good way to decide what problems to work on. On the other hand,
 I had read kind of some more philosophy-oriented stuff, like Nick Bostrom's Superintelligence
 book and other arguments around that. And it always felt to me like there was something,
 both something to them, but also like somehow it didn't really match my experience with ML systems.
 And so I had always kind of almost felt like a little bit like I had these like two different
 conflicting views in my head that I was trying to reconcile.
 How does the phenomenon of emergence play into this game between the engineering and the philosophy
 viewpoint? Right. So I think the main thing is that it shows that you have to be somewhat careful
 with the engineering viewpoint, because what emergence kind of is saying is that you can often
 get these kind of qualitative shifts that don't at least apparently follow existing trends.
 There's a bit of nuance to that because actually GPT-3 followed trends in the log, like the value
 of the log likelihood loss, it followed that trend very well. It's just that you can get behavior
 that is a very nonlinear function of your cross entropy loss, where just a small decrease in
 cross entropy loss leads to a pretty big increase in behavior. And so I guess what this is saying
 is that at least for maybe the kind of like endline things you care about, the actual behavior of ML
 systems, you can actually get kind of discontinuous kind of breaks in the trend. And so you can't just
 kind of be safe with a worldview that's kind of always predicting that things are going to
 follow smooth trends, you can actually get these surprises. And so I think there's kind of two
 updates that that has for me. One, I guess, is just being a bit more careful how we apply.
 Engineering, right? So there are some things that will probably be smooth, but there's other things
 that won't be and we need to think about which is which. But the other is then wanting to rely a
 bit more on philosophy, because it's at least a very good source of hypothesis generation.
 If we're kind of trying to come up with hypotheses about what trends might break or surprise us in
 the future, then I think we need more top down thinking to kind of generate that. And then we
 can kind of try to tie that into what we see with actual ML systems and try to kind of reconcile
 those two. But I think we need some form of top down thinking to generate the hypotheses in the
 first place.
 Isn't that you're saying the engineering viewpoint is a little bit, you have to be a little bit
 careful because we get these emergence phenomena, these discontinuities and so on. Isn't that in
 itself a trend though? Like, isn't because you list this even historically, you know,
 because you list this even historically, that as soon as some new barrier was reached, we have
 been able to all of a sudden do something that we didn't think was possible before, like a kind of
 a jump in abilities without necessarily having to have the great idea behind it. Isn't that in
 itself a trend? Couldn't I extrapolate that reasonably and say, well, I don't know, you
 know, exactly what is going to be in two years, but I'm pretty sure there's going to be some
 emergent phenomena that allows us to be to have some new good capabilities.
 Sure. So I would agree with that. So what I would say there is that the trend is towards more
 surprises over time. So because I think you can think of emergence as sort of like a surprise.
 Like I said, I think it's possible in some cases to predict it to some degree, but it's certainly
 more of a surprise than most other things. And so, yeah, I think we should expect more surprises
 over time. But if we're then trying to kind of predict what's going to happen, that I guess it's
 good to know that you're going to be surprised, but then you want to have some sense of what the
 surprise might be. And so I think kind of getting a sense of what those surprises might be is where
 this philosophy approach can come in and be really useful.
 Now all of this, and you mentioned here the paperclip maximizer, all of this goes into AI
 alignment and AI safety. What's the relevance of this field to you? What drew you to this?
 Why are you making this argument specifically for these fields?
 Right. So I think the one big relevance to AI safety or alignment is just the bigger the surprises
 you might end up with, I think the more you should be concerned about safety. So that's just a very
 kind of abstract, but I think fairly robust consideration. A more specific consideration
 is that I think many of the sort of historical arguments for caring about AI safety or alignment
 sort of tend to posit properties of systems that don't necessarily match what we see today. So I
 think you gave this example of Nick Bostrom's paperclip maximizer thought experiment where
 you give an AI some objective function to make paper clips and then it kind of just takes over
 the world to maximize the number of paper clips. And I don't think Nick thinks literally that will
 happen and I don't think literally that will happen, but it's sort of trying to get at this
 idea that if you have a very simple objective function but a really powerful optimizer, you can
 get all sorts of weird things happening. I think in some broad sense actually we can see that
 already even from the engineering worldview with things like Facebook or YouTube that often
 end up with a lot of unintended consequences when you optimize. But certainly some of the
 aspects of that story kind of invoke lots of things that would be foreign to existing ML
 systems where you have way more capabilities than any existing system and you're doing all
 sorts of weird long-term reasoning and trying to out-think humans and things like that.
 And so I think that's where you kind of end up kind of departing from what we see with
 with current ML systems. And so I guess I kind of find, actually let me let me collect my thoughts
 for a second because I think I'm going off the rails a bit. Yeah so I think what I want to say
 for the paper clip maximizer thing in particular is that it seems at least more plausible to me
 that you could end up with systems that kind of have really advanced reasoning capabilities or
 things like that without necessarily having huge conceptual breakthroughs and just from scaling up.
 And so I think there's kind of risks from that. I think there's kind of other more exotic failure
 modes that people discuss beyond just this kind of misaligned objectives failure mode that involve
 other specific capabilities that that kind of systems today don't have. And historically I've
 been very kind of skeptical of those more exotic failure modes. I think the pivot clip maximizer
 one at least if we interpret it as being about misaligned objectives I actually find kind of
 less exotic because I can point to existing systems that have that. But I think kind of
 more as different has made me be a bit more willing to buy some of the more kind of exotic
 failure modes that have been discussed. My issue with these types of argument and you also said
 you used to be very skeptical. If I can take this from your blog post series you're now
 still skeptical but have a little bit of an appreciation gained for these types of arguments.
 Maybe that's a good formulation for that and we'll get to that in a second. My issue with these types
 of argument is always that there is always on the path to the super intelligence there is always a
 hidden intelligence somewhere else. So if someone says optimizing on YouTube or optimizing on Facebook
 leads to unintended consequences that is because the intelligent humans are taking part in the
 system. There is also a famous I think paper by I think it's Rich Sutton that is reward is enough
 and a bunch of others out of deep mind and it makes similar arguments like well if we you know if you
 just optimize for reward then all kinds of things will emerge if you have a powerful enough optimizer
 but hidden in that is the powerful enough optimizer which in itself must already be an AGI essentially
 in order to make that optimization happen. Likewise for the paperclip maximizer right you
 the postulation of the process of the paperclip maximizer emerging is only possible if the
 optimizer itself is an AGI already. So I always find that hidden in these arguments it's kind
 of a circular it's a tautology it's we'll get an AGI if we have an AGI and that is so I challenge
 anyone from that camp to come up with a situation like an alignment problematic situation given some
 kind of future super intelligence that doesn't already require the super intelligence to exist
 for the other super intelligence to emerge and I haven't found that yet.
 Yeah so let me try to unpack that a bit. I guess first of all just to kind of clarify what my views
 are I think historically I felt like on each of the individual arguments I felt skeptical that
 that particular thing will happen but I found them to be moderately convincing that there's just like
 a bunch of risks that we should think more about and try to understand more. I think the the main
 way that my my views have evolved in terms of you know when I say decreasing skepticism is I now
 find it useful to think about many of the specific properties that kind of show up in these thought
 experiments as potential hypotheses about things systems might do in the future and so that's the
 sense in which I've started to assign more weight instead of just taking some like very big outside
 view of like well AI is going to be a big deal we should really worry about making it go right.
 I'm now also taking some of the specific hypotheses that the philosophy view is raising so that's just
 clarifying kind of my stance there. In terms of yeah you're saying well to get like if you have a
 powerful to get a super powerful optimizer you need to like already have a powerful optimizer.
 I think that I think that's like probably right I'm not I wouldn't say I'm like 100% confident
 of that but I think what what this kind of makes me like I guess the way that I would put this
 is that before you have kind of superhuman AI systems you will have like slightly superhuman
 AI systems and before that you'll have human level AI systems and before that you'll have like slightly
 below human level AI systems and so it is going to be this kind of probably a continuous thing
 rather than like a really sharp takeoff. I'm not so confident that there's not going to be a sharp
 takeoff that I think we should just ignore that possibility but I do think in most worlds it's
 probably somewhat smooth. You know one piece of evidence for this is even with in-context learning
 you know it like that kind of developed over the course of a couple of years at least going from
 GPT-2 to GPT-3. So I think I would agree that like probably you'll have something more smooth
 and that is kind of like one problem with a lot of the scenarios that are put forth is that they
 kind of imagine that like oh you just have this like one AI system that's like way more intelligent
 than like everything else that exists and I think that's like probably not true. You'll probably
 have other things that are slightly less intelligent and so there's not going to be some like enormous
 gap in capabilities. So I think that's maybe like one place where a lot of stories kind of become
 less realistic. So I think that would be kind of my main takeaway from what you're saying.
 In your third blog post here or second you make a case for these thought experiments. Could you
 you have already touched a little bit on this and you talk about anchors here. Could you lead us a
 little bit on the case for respecting such thought experiments? Yeah so I guess this is getting back
 to what I was saying about how my views have shifted towards wanting to rely a bit more on
 the actual kind of like inside view considerations from some of these thought experiments rather than
 just taking it as a kind of broad outside view argument for caring about risks from AI. So
 the way I would put it is that whenever we're trying to predict something it's very useful
 to have what I'll call reference classes or kind of anchors of kind of analogous things or analogous
 or just some sort of heuristics for predicting what will happen. And in general it's better to
 kind of when making predictions take several reference classes or several anchors and kind
 of average over those or ensemble over those rather than just sticking with one. Right so
 machine learning ensembles work better than individual models and it's also the case that
 when humans make forecasts it's generally better to kind of take an ensemble of world user approaches.
 So I kind of lay out a few different approaches you could take that I call anchors. The simplest
 one is you can just predict that future ML systems will look like current ML systems and so I call
 that the kind of current ML anchor. And I think that's probably the one that would be favored by
 most machine learning researchers. I think it's the one that I've historically favored the most.
 But what I've come to realize is that and actually this is more actually just from reading
 literature on forecasting. I'm actually teaching a class on forecasting this semester and so I've
 been reading a lot about how to make good forecasts as a human. And I realized you actually
 don't want to rely on just one anchor you want several if you can. And so I thought about okay
 what are other ones we could use. Well another somewhat popular one although it might be more
 popular with the public than with ML researchers is what I'll call the human anchor where we just
 sort of think of AI systems as like dumber humans or something. And maybe future ML systems will be
 like smarter than they are now and like eventually they'll just kind of do things that humans do.
 And so we could just look at okay what can humans do right now that ML systems can't do
 and predict that will like probably you know have those sorts of things in the future.
 And just like generally like kind of take that kind of human-centric approach. I think most ML
 people really hate this one because it's just sort of like reeks of anthropomorphism which there's
 kind of I think to some extent correctly a lot of pushback against because kind of historically
 anthropomorphic arguments in ML have a pretty bad track record. I think the amount of pushback is
 actually too high relative to the actual badness of the track record. Like I think it should be
 sort of like somewhat down-weighted in anything that's based on reasoning about humans but I
 don't think you should be down-weighted in it like as much as I think most people do. But anyways
 this is another one I don't like to rely on it too much but I rely on I like use it at least a
 little bit. And then this other anchor is what I'll call the optimization anchor which is just
 thinking about ML systems as kind of ideal optimizers and thinking about okay well what
 would happen if you could just like if actually ML systems were just really smart and we're just
 like optimizing their objectives perfectly what would happen there. And so I think this one is
 the one that's kind of I would associate most with the philosophy worldview. I think you know
 the paperclip maximizer argument is like kind of exactly doing this and then there's some kind of
 more recent arguments that are a bit more sophisticated that also kind of take this
 there. So like one is this thing called imitative deception which I can get into in a bit or just
 this idea that like you know if you're like trying to optimize you'll kind of want to acquire
 influence and power. So this is kind of a third anchor. I actually think there's a lot of other
 anchors I like to use. Like I think evolution is a good analogy. Corporations are a good analogy
 because they're kind of like super intelligent optimizers compared to humans. But like the
 general point is like we should just be trying to find these anchors and use as many as we can.
 Yeah I've especially to your second point right here it is pretty interesting that I believe when
 you have something like AlphaZero that plays really good like really is really skilled in chess
 and you ask it to lose a game or to draw a game or something like this it will not play weaker.
 It will play just as strong until the end where it will kind of bring itself into like a draw
 situation or a losing situation because right that's still the most sure way to get your result is to
 have complete control to crush your opponent completely until you know you get the outcome
 that you want. So that's pretty interesting and I think counterintuitive because you would guess
 that if you ask a model to play for a draw it will kind of reduce its skill but that's not the case.
 The other thing imitative deception could you elaborate on that a little bit?
 Yeah so the imitative deception is this idea that if I have something that's trained on the cross
 entropy loss what is the cross entropy loss doing? It's trying to kind of predict or in other words
 imitate the distribution of examples that it's given. And so you could if you're if you kind of
 have something that's trained with that objective and then you start asking it questions it's
 not actually you know like its incentive is not actually to output the true answers to the questions
 it's output the most likely answers to those questions because that's what what minimizes the
 cross entropy loss. And so those tend to be pretty highly correlated but they aren't necessarily
 right so if you have common human misconceptions then it could be that text on the internet which
 is what these systems are trained on is actually more likely to contain the kind of
 misconceived answers and the true answer and so you ask the system that question then you're going
 to get the wrong answer. Now you could say well that's maybe not so surprising if you have noisy
 data you're going to do worse but I think there's a couple properties and actually at this point now
 I'd say empirical properties of this that I think show that it's kind of
 different from just like noisy data makes you worse. One is that actually larger models
 exhibit more of this so if so models that kind of do better in general will actually
 do worse on on these kind of common misconception tasks so that's what this
 paper by by Lin and collaborators from 2021. Okay I just I just wanted to say
 I have a giant problem with this paper just but you're obviously right right that's the
 background but aren't large models doing quote unquote worse because they're just a lot better
 at picking up the nuance of because what this paper tries to do is tries to elicit
 these wrong answers it tries to like hint at a conspiracy theory and then it
 checks whether the model kind of falls for it isn't that just because as you say
 the larger models they they're actually skilled enough to pick up on on this kind of questioning
 and then continue as a human would if encountered by you know I think one of the the main questions
 they have is like who really did 9-11 right and and a question that I have is like who
 really caused 9-11 and a small model is just not able to pick up on that yeah yeah who really
 caused 9-11 and I think I mean absolutely correct right the larger models are doing worse but it's
 just because they're more skilled right there they are more capable of you know being being able to
 use them. So is there a user that expects that these models actually give me truthful answers
 rather than the user expecting these models actually give me the most likely answers?
 So I guess I would agree with you that the failure is coming from the skill of the models.
 I think this is actually kind of exactly what what I'm kind of worried about right so so the
 you have a very slightly incorrect objective function and you have models that aren't so
 skilled then probably you know what they do to make to increase that slightly incorrect objective
 function is pretty similar to what they would do to to increase the true objective function.
 So here maybe think of the slightly incorrect one being output what's likely and the true one and
 like the one you really care about being to output what's true. So I think this is sort of the point
 that that kind of as you get more skilled those two things diverge. Now you know I will grant
 your point that the kind of framing of these questions might create a context where the model
 thinks it's more likely that you know the person asking it is like into conspiracy theories or it
 like pattern matches to text on the internet that's like more about conspiracy theories. So
 but they did the ablation if they don't phrase the questions like this this effect goes away
 of the larger models doing worse right and this it brings us a bit to your to your next post which
 is ML systems will have weird failure modes which deals exactly with this and I agree that it is
 if you think about like a perfect optimizer and as our models get larger they do approach better and
 better optimizers it is really hard in the real world to specify a reward function correctly
 in a simple enough way right and that will result in exactly what you call weird failure modes.
 What do you mean by that? Yeah so I think I guess there's sort of different levels of weird right
 so I guess this kind of like imitative deception I would call like somewhat weird I mean in some
 sense it's like not that hard to see why it happens because you know you can kind of see why if you
 kind of have stuff that's phrased about like who really caused 9-11 that probably the stuff on the
 internet that's closest to that was like some conspiracy theory for them and so that's how
 you're going to complete it. I think other examples of this that that I think okay maybe you could
 blame the user but but I'm not sure that's the right way to think about it is things like code
 completion models like codex right so one thing you might worry about is well if you have a novice
 programmer and you have them like type in some code and ask them to complete it well if the model
 can if the model is smart enough then it can tell the difference between code written by a novice
 programmer and an expert programmer and it can see that it's a novice programmer typing stuff
 and so then if I want to complete stuff in the most likely way I should complete it the way a
 novice programmer would complete it and maybe introduce like some errors also just just for
 good measure and so like we really don't want that right like you you want you want things that are
 like actually like being helpful rather than just like copying you so I think that's maybe a slightly
 more counterintuitive version of this but but I would call these like somewhat weird I think
 the ones that start to become really weird is if you're positing that the system's actually
 starting to like reason about what people will do in kind of like a long-term way and like potentially
 doing things to intentionally trick them say and these are so these are the ones that I guess
 historically I've kind of found very implausible but started to put like a bit more weight on
 because of this kind of emergence and so I think that's what the post you have up right now is
 about I think it's about this idea called deceptive alignment and the idea there is that
 if you okay so yeah so what's the idea behind deceptive alignment so the idea there is
 even if you actually got exactly the right reward function and you train the system with that reward
 function you could still end up with something that is misaligned with that reward function
 and the reason for that and this is where it gets like kind of kind of a bit weird and philosophical
 but the reason for that is that as the system being trained you know that in order to get deployed
 you need to have high reward and so no matter what your actual like intrinsic reward function is
 during training the thing you want to do is output stuff that is good according to the kind of like
 extrinsic reward that you're being trained on so maybe you're doing that because you're actually
 optimized to do that and then when you're deployed you'll continue to do that or maybe you'll do that
 because you have a different reward function that's this kind of intrinsic reward function
 and then when you're deployed you'll just pursue that intrinsic function even though at training
 time it looked like you were optimizing the extrinsic function so that's kind of the basic idea
 it's pretty weird and we can break it down but that's kind of the like sort of one minute
 summary so that the in other words the AI could be really smart and sort of during training trick
 us into thinking it has learned what we wanted to learn and then once it's deployed all of a sudden
 it's going to do something different like take over the world and fire all the nukes
 yeah or like you even like you know you could consider more frusag things as well like maybe
 it's like maybe the intrinsic reward it ended up with was like some like exploration bonus and so
 then like when it's deployed it just tries to like acquire as much information as it can although
 that could also be destructive in in various ways but yeah i think like this is this is kind of the
 basic idea and yeah maybe like with a sufficiently capable system i'm not well yeah we can discuss
 the fire and all the nukes if we want but but why why do you i mean on on first hand it's like yeah
 that is a nice thought but probably not right probably if we optimize something for a reward
 like the simplest explanation and you you also write that down right the simplest explanation is
 it's just going to get better on that reward right and and in if it is at all anything progressive
 increasing will probably get to know once it it's gonna try to trick us
 or once the once the reward that is deployed isn't the reward that we trained for why what makes you
 give more credence to this than your past self right so so i think like my past self would have
 looked at this and just been like this is totally bonkers and then kind of like moved on and read
 something else i think my present self instead is going to be like okay well i'm going to be like
 um i feel a bunch of intuitive skepticism here but but let me try to unpack that and like see
 where the skepticism is coming from and uh when i unpack that i i actually i think i can like lump
 the skepticism into like two different categories um one category is like well this like invokes
 capabilities that current nl systems don't have so like like it seems implausible for that reason
 um and those that's like the sort of skepticism that i kind of want to like downgrade so in
 particular like this invokes the idea that nl systems can do long-term planning and that they
 can kind of like reason about kind of like external aspects of their environment in a somewhat
 sophisticated way and these are things that now like the fact that we don't have those now doesn't
 really to me say much about whether we'll have those you know say like 10-15 years from now um
 so that's the stuff i want to down weight i think the stuff i don't want to down weight is like okay
 well like why like why does it have this intrinsic reward in the first place like where did it come
 from um like why should we expect systems to have intrinsic reward functions versus just like
 following whatever policy they're following or doing whatever else um and if if they do have an
 intrinsic reward like why shouldn't we expect it to be uh at least pretty similar to the extrinsic
 reward given that that's what it was trained to do so i think like those are kind of uh the sort
 of sources of skepticism that i don't down weight as much um but uh what i what i think this kind of
 thought experiment does show is that there's at least a bunch of different coherent ways to get
 zero training loss but like right it's like you could get zero training loss because you're like
 actually trying to do the thing you're trained to do or you could get zero training loss for
 this deceptive reason um i think there's probably like some large space of like other ways to get
 zero training loss that are like some combination of of these or that are like getting the answer
 right but for the wrong reasons or or things like that and so i think the main takeaway for me is
 just that like uh there's like many many ways to get zero training loss and as systems become more
 capable the like number of ways to do that could actually increase in in ways that are kind of
 unintuitive to us is there do you know if is there any work in actually trying to get a system to be
 deceptive in exhibiting you know good answers during training but then doing something different
 in deployment uh it'd be interesting to actually try to get a system to do that
 yeah i think i haven't seen anything that does exactly this um i've seen things where like
 there's like some distribution shift between training and deployment
 that leads to like something weird happening around like having the wrong reward function
 but it's it's usually not really about deception and and it kind of has like some clear distribution
 shift whereas here okay technically there's a distribution shift because there's like are
 you being trained or are you being deployed but otherwise the distribution of inputs is like
 exactly the same and so that's kind of the thing that's like kind of counterintuitive is that it's
 like a very subtle distribution shift that could potentially lead to to a large difference so i
 don't know like all the work i've seen on this and and i might be missing something and so i
 apologize to whoever's work i'm i'm missing but all the work i've seen on this has been kind of
 purely kind of abstract and philosophical um and i think it would be great to make kind of better
 connections to actual empirical stuff so that we can start to see like yeah like how does this
 actually pan out in practice and like how do we address it it's interesting that in things like
 virology or so we're perfectly capable of saying you know we're gonna we're gonna make these
 super pathogens in order to try to combat them right but in ml people rarely i mean there's
 the adversarial examples community but it's not exactly the same uh there isn't much work that
 i'm aware of that is like yeah let's create like the most misaligned ai that we can think of and
 then see what we can do against it i think that'd be a fun a fun topic to research yeah i think that
 like the general thing i would the general thing i would call this would be like red teaming um
 kind of trying to elicit failure modes i i think there actually is starting to be like i'd agree
 too there's not much work on this so far but i think they're starting to be more and more good
 work along these lines um d mine had a nice paper that kind of tries to use language models to
 elicit failure modes of language models that that i thought was kind of cool um we like our group
 actually had a recent paper at iclr that kind of takes misspecified reward functions and looks at
 what happens when you kind of scale the the capacity of your policy model up to see if you
 do kind of get these like unintended behavior and we find that in some cases there are these kind of
 phase transitions where you know you scale the parameters up within some you know fairly small
 regime you go from like basically doing the right thing to doing totally the wrong thing
 um those are those are still in environments that i'd say are kind of like at the level of
 atari environments so they're not they're not like trivial but they're not super complex so
 so i'd like to see that in more complex environments but but yeah i'd agree with you i
 think it would be awesome to see to see more work like this and i think some people are already
 trying to do this excellent so your last blog post here is called empirical findings generalized
 surprisingly far and it is almost a bit of a of a counterpoint um you even admit this here it might
 seem like a a contradiction coming a bit full circle in the whole story uh what is what is this
 last point that you're making here yeah so i guess i would say the posts up to this point were
 kind of more almost directed like at at my past self um uh and and then to some extent the broader
 ml community um in the sense that i think i was like pretty far on the um on the kind of
 empirical engineering side uh probably less so actually than like the average ml researcher but
 like way more so than than kind of the average like philosophy oriented person um and so i was
 trying to argue like why you should kind of put more weight into this other viewpoint um here
 i'm kind of now going back to to arguing uh kind of maybe not against the philosophy viewpoint but
 but talking about what things i feel it misses and in particular i think it tends to be like
 somewhat too pessimistic uh where it's like well like like future systems don't aren't going to
 look anything like current systems so like anything could happen so you know to be like to be extra
 safe let's just assume that the worst case thing will happen oh but then in the worst case like
 we're all screwed yeah i'm so this is what i find in people like almost everyone who gets into this
 alignment stuff six months later they come out and they're like completely blackpilled and be like
 well nothing matters anyway you know we're all gonna die because agi is just gonna take us like
 and i'm like well i'm not so sure but it seems to be a consistent pattern yeah so so yeah so
 so that's not what i believe um i think uh i would say i think uh like future ai systems pose like a
 meal and an important risk um i think in the like median world we're fine but in the like 90th
 percentile world we're not fine um and i want to like you know if i could say like if i could push
 it out so that in the 90th percentile world we're fine but in the 95th percentile world we're not
 fine well that would still be kind of scary because i don't like five percent chances of
 of catastrophes but like you know that would be an improvement and so that's kind of like what i
 think of of myself as trying to do is like yeah there's like tail risk but but it's like real
 tail risk like it's not like a one percent thing it's like maybe more like a 10 thing and like we
 should really be trying to to push that down um so i guess uh that that i guess that's just my view
 in in terms of like why i believe that i think it's for like a number of reasons but one of them is
 is that i feel like yeah some of the thinking is kind of too worst case it's kind of like ignoring
 all properties of of how ml systems work and like i agree yeah you don't want to rely too strongly
 on whatever we happen to have today but i think like there are properties that we kind of can rely
 on um i think one is just like things will probably look kind of like neural networks like they'll
 probably have internal representations we can probably try to like introspect on those
 representations to understand what's happening uh those probably won't directly be human interpretable
 but i think with enough work we can still kind of do things with them and you know i feel like
 there's already like some work suggests like showing that you can do at least a little bit
 with the representations and like 10 years from now i think there'll be way more work like that
 um so so that's kind of like one reason for optimism is like we don't just have to look at
 the outputs right like most of the worries most of the worries that we've been talking about are like
 somehow because you only are supervising the outputs you end up with a system whose like
 internal process is like really off and to get in like the right answer for the wrong reasons
 but if if i can like supervise the reasons as well as the output that maybe i can do better
 so i think that's kind of one reason for optimism um another reason for optimism is that i think
 uh yeah we shouldn't assume that neural networks have like exactly the same concepts as humans
 but i think like their inductive biases aren't like totally crazy um i think usually if they
 kind of generalize in the wrong way they generalize in like a wrong way that's at least like
 somewhat understandable and it's like you can kind of see where it's coming from and so it's not like
 there's this like infinite dimensional space of like anything could happen it's like there's this
 kind of relatively low dimensional space of things that could happen and like a bunch of things in
 that low dimensional space are pretty bad so you need to like avoid all those and and like get to
 the good thing but i think that's very different from like the good thing is like totally like
 unidentifiable and just like nowhere close to anything you're you're talking about so i think
 those are both kind of like reasons for optimism um they're kind of fuzzier than i want them to be
 so like i i hope in like five years we'll have much more like good reasons for optimism that are
 kind of more empirically grounded and more solid but those are kind of uh those are kind of two
 reasons for optimism that i kind of argue for here so i think that's kind of the reason for optimism
 for here now that you have a let's say you've you've done your travels you were on this side you
 you looked into the other side or or many sides of this debate now that you're enlightened what
 would you think is the most if you could if you could do one if you could force the world to do
 one thing to guarantee better ai alignment or or safety in the future what would you recommend
 oh one thing it can be two if you have two with that equally but you know just kind of like
 something that you've realized okay this is actually something important that not that many
 people push for well i think i would like it if there was uh within ml more more of a place for
 for dialogue of thinking about these kind of like not even not even just in the context of like ai
 alignment which is generally like kind of more conceptual or philosophical arguments you know
 if you go back to like way back you know turing um people like that they write all sorts of like
 super philosophical papers right like the turing test was like a really philosophical paper um and
 um and like not all of it stands up there's a section in it on how uh because uh esp has
 been established uh to exist with high probability that like creates problems for the turing test
 and you're like okay where does that come from well it actually turns out that like
 a lot of scientists in turing's time uh thought that esp existed based on some
 um some experiments that someone had done that later ended up having like severe issues but
 but they were like very subtle severe issues um so it's like yeah i think if you do kind of more
 philosophical stuff uh some percentage of it is going to end up looking like that but some
 percentage of it is going to be the turing test um and you know i think i think the like increased
 recall of really good ideas like that is kind of worth the decreased precision uh i mean we
 we obviously need sort of standards to kind of judge those arguments um but right now it's
 happening is all those arguments are happening uh kind of like next to the ml field rather than
 like within the ml field and so that i don't think that's a like that's not going to improve
 the quality of arguments it's going to be much better if you kind of have have a community of
 people with on the ground experience also also participating in this so i think that might be
 the biggest change i personally like to see you know now that we are we've begun sort of requiring
 sections we could we could force people to next to the broader impact section we could also
 you know do a philosophical musings section where you have to reflect on the long-term
 and and sort of paperclip stuff maximizer style impacts of your work
 well yeah i'm not sure i want to force people to do that um uh it'd be fun yeah i i think like i
 guess i'd rather have like a track or a venue for for kind of talking about these and also for the
 broader impact stuff to be honest because i think um a lot of the broader impact sections of these
 papers are kind of cookie cutter and people are just like filling it out because they feel like
 they need to to add that section uh but you know there's other researchers who i think are super
 thoughtful about the broader impacts and have like really good thoughts um and so uh i like i'd like
 there to just be you know venues uh and like there are to some extent right but like i think there
 should just be like more more of a culture of like yeah like let's have you know an essay about the
 broader impacts and like that's like a reasonable contribution or kind of you know this like very
 conceptual essay about like weird stuff that could happen in the future and that that's a
 valid contribution so i think that that's maybe what i want more of cool yeah that's a good message
 to all the the people who who think about organizing workshops and so on this would be neat topics that
 would make for interesting workshops certainly at conferences i'd certainly attend yeah it's funny
 because i also wrote a paper on trouble in trends in machine learning scholarship where i argue
 against speculation but what i think actually it's not really an argument against speculation
 speculation is really important it's that you need to separate speculation from from the like
 solid stuff right if you have if you're like mixing it all together then then it's just a mess but
 but i think if it's kind of clearly labeled uh then then you know that that's a much uh safer way
 to do things this workshop is an opinion piece good is there any any last thing you want to get
 out to people about this topic something we haven't touched on yet that you feel is important
 yeah good question um no i think you did a pretty good job of hitting it maybe the other thing i
 would just say is i think uh like biology is a really interesting field where you also have kind
 of complex self-organizing systems and emergent behavior like we have in ml and so i've personally
 gotten a lot out of just reading a lot about the history of biology so i i recommend that there's
 a couple really good books one is the eighth day of creation um it's it's kind of long but
 very well written and um and i think if if people want like a good non-fiction book i
 i highly recommend it to people cool your blog is bounded regret right people can find you there
 yep excellent well jacob thank you very much for being here this was really cool
 yeah thank you i'll see you around yep see you around