ROME: Locating and Editing Factual Associations in GPT (Paper Explained & Author Interview)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper
#ai #language #knowledge 

Large Language Models have the ability to store vast amounts of facts about the world. But little is known, how these models actually do this. This paper aims at discovering the mechanism and location of storage and recall of factual associations in GPT models, and then proposes a mechanism for the targeted editing of such facts, in form of a simple rank-one update to a single MLP layer. This has wide implications both for how we understand such models' inner workings, and for our ability to gain greater control over such models in the future.

OUTLINE:
0:00 - Introduction
1:40 - What are the main questions in this subfield?
6:55 - How causal tracing reveals where facts are stored
18:40 - Clever experiments show the importance of MLPs
24:30 - How do MLPs store information?
29:10 - How to edit language model knowledge with precision?
36:45 - What does it mean to know something?
39:00 - Experimental Evaluation & the CounterFact benchmark
45:40 - How to obtain the required latent representations?
51:15 - Where is the best location in the model to perform edits?
58:00 - What do these models understand about language?
1:02:00 - Questions for the community

Paper: https://arxiv.org/abs/2202.05262
Follow-up paper on Mass-Editing Memory in a Transformer: https://arxiv.org/abs/2210.07229

Abstract:
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at this https URL

Authors: Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, today we're talking about locating and editing factual associations in GPT by Kevin Meng, David Bao, Alex Andonian and Yonatan Belenkov. In this paper, the authors attempt to localize where in a forward pass through a language model an actual fact is located or where it is realized. For example, something like the Space Needle is in downtown Seattle. It has a subject, a verb and an object. And where exactly in a language model does the language model know, quote unquote, these things and that the Space Needle is in downtown Seattle? That's the question of this paper. And they go beyond that by figuring out where these facts are. They can also then edit those facts, meaning they can change the model such that it all of a sudden believes that the Space Needle is in Paris. And they test in various ways that this change is first of all robust, it generalizes, but it doesn't distort the rest of the model too much. Moreover, this change is like a rank one update that they can pre compute. So all of this is very, very interesting. And we're going into it in detail. This video is a bit of a mix between me explaining the paper and the authors with whom I interviewed, giving their inputs into various aspects of these questions. I hope this is of benefit to you. Let me know if you like it or not. And let's go into it. There's an entire subfield that just researches where are facts in language models. I didn't know about the subfield until I read your respective works. What does it entail? What are people wondering about? So I guess there's a few questions. I think it's at the intersection of two main things. One is a scientific investigation into where things are and what models are doing to achieve them. And then at the other end of the spectrum is a practical question that sometimes these models mess up. Because they have information that we want to change because it's now outdated. And how do we do this in a practical, in a very clean way? On both sides, there are individual respective questions. On the interpretability side, I think David might be able to talk about it a bit because he's worked with not only language but also vision models. But yeah. Yeah, so I can talk about the interpretability side. Sounds good. So on the interpretability side, it's this really old question that's gone back to sort of the early days of neuroscience. Where do ideas and where does knowledge live in a big neural network? People thought about this in the biological neural networks of your brain. There's this old theory of the grandmother neuron that maybe you could even have a single neuron that's responsible for what you think of your, for thinking about your grandmother. Maybe if you pluck that neuron out of your brain, you might forget that whole concept, which people think is sort of implausible. But what we're chasing here is sort of a weaker locality question. Like, if you have some knowledge in a big neural network, can it be localized to a small set of neurons or small set of layers? Can we find out where that knowledge is? And so there's been a bunch of people who have been looking at this. It's, you know, I guess maybe the overarching area is called like mechanistic interpretability research where people are trying to understand the mechanisms that are emerging inside the learned computations. And so there's, there was a really nice paper by Al-Haji from, from Anthropic. There's been a series of papers from, from JIVA, from, from Israel, who've been looking at the structure of computations inside the network. And so our paper is another contribution in this direction. I think the thing that we're looking at a little differently is we're using, we're really focusing on using causal probes to ask that question, you know, making changes in the network to see how the network responds when we make changes and using that to map out things. And what I, what I love about your work is then you actually put it to the test, which means that if, if we understand where the knowledge is, we should be able to change it, right? And that gives to me, the interpretability research is always a bit shrouded in mystery because there are always, I feel something like 10,000 different explanations that could explain a given fact. And usually the researchers frame it in a way that their hypothesis makes the most sense, but I'm always like, meh. But if you then actually put it to the test and you say, well, if we are correct, we should be able to edit the knowledge, we should be able to erase a factor, insert a new one using what we think happens. And that's also a thing that you do very well. Yeah. So I think that's where the really interesting interplay between the interpretability and the practical side comes in, because on the practical side, people have been chasing this question of, of, of, of real world usage. Like these models are huge. They're really difficult to retrain. And then when we actually do fine tune them, for example, on a small data set with a, with sort of a blind objective, it's kind of hard to tell sometimes what we're doing with it. And so in the past, we've seen some works, for example, from Mitchell and from Decau. They spent a lot of time asking the question, like, can we achieve generalization when we do edits? When we change one thing, does something else change? Or is the edit specific? Like if we change one thing, does an unrelated fact also change undesirably? So they've kind of set this area up because it's a very practical question. And I think the really cool thing about Roam is that, like you said, on one side is the scientific question, but on the other side, we show that the insights that we get can yield a pretty useful model editor that seems to achieve generalization, specificity, and fluency preservation all pretty well. I was wondering since, since the main foundation of neural networks is distributed representations, this is the big step, right, to go from go-fi systems, from symbolic systems to distributed systems where we no longer have individual symbols representing individual things in the world, which we could build, you know, very simple knowledge graphs. Now a fact like the space needle is in downtown Seattle needs to be stored somewhere in a vector space. Yet you managed to actually locate that fairly well to particular points in the network. How does, how does that work? So here is how causal tracing works. This is one of the main methods the authors employ to figure out where in the model the facts are realized. We are talking here about the realization of facts, which is connected to the storing of facts, but we specifically care about the activation, so the hidden signals as they travel through the networks and not necessarily localizing facts inside of the weights of the neural network. So in this case, you can see that here is a sentence that you input, the space needle is in downtown and the model would output, well in this case, it's an uncorrupted sentence, the model would get this correct. If it's a good language model, you'll get this correct to say Seattle as the next token. This as you can see goes through a number of different stages. So due to how GPT works, how a autoregressive transformer works with causal masking, you will have the word, the token for the being embedded, generating a hidden state here. Now that hidden state, first of all, it goes through essentially the layers of the transformers and it accumulates two things. So it always accumulates an attention head and it accumulates a multi-layer perceptron head, or actually, I think two in succession, and then there's a residual connection around that. So that's what you see right here. But also the same hidden signal on each layer travels forward essentially. Well, not exactly, it's more like when the second token or the third token, when they come in, so when space is now fed into the transformer, it now gets a signal from the past, because it does causal attention, it looks at the past. So it also will get kind of the hidden signals, the hidden states from the past. So essentially this would flow like so, but every time it would also get the hidden signal from there. And then need will get the hidden signal from both the and space, so it would get both of them right here, but also it would travel up the layers and get both the hidden signals from here. So you can see there is various paths this information can take. And the idea here is to figure out where in these hidden states, so in these bubbles right here, or this bubble, or this bubble, where is the fact that Seattle should be the output of the sentence? Where is that kind of realized? Where is that localized? Now you might have various opinions where that's localized. First of all, opinions here, like where in the sentence does the model kind of put a lot of weight on Seattle and where in the network? So here in the depth of the network, where does that happen? And both of them, what turns out as evidence, both of these things are quite surprising. So here what they do is this causal tracing. What they do is they run the model once with a clean input. They record all of these hidden activations. Then they run the model again, but this time with corrupted input. So here you can see these have little asterisks by them, which means that the input is now corrupted. It means you add some noise or you just replace them by noise or replace them by something else. It's just not the original signal anymore. And therefore, if you just let the model run, it will probably produce something else because the subject, so this is the subject of the sentence, is completely corrupted. So this could be whatever is in downtown. And then Seattle is certainly not the first thing on the model's mind. It might be, but it's like very likely not. And then what they do is really interesting. They now take each one of these things here individually. They take a hidden state and they just copy it over. They just copy that over. So instead of at this particular hidden state, instead of what the model gets as an input, you know, from this path and from this path and from this path here, instead of that, it just ignores that particular hidden state and replaces it with the one from the clean input. And now we observe, so here maybe it said like Paris before because something is in downtown, the model just said Paris. And now we observe, if it kind of stays at a wrong answer, then that hidden state, that original hidden state was probably not super well associated with either the input space needle or the output Seattle. However, if copying over that hidden state from the clean signal actually changes the output back from Paris to Seattle. Well, that is a fat marker, oh, sorry about that. Those are my notes. If that actually changes it back, then we know, aha, this hidden state must be quite important for sort of associating space needle to Seattle. And that's how we find out. And as you can see in the results, you get these two clusters, you get an early, what they call an early site, which usually happens after the subject is done, and a late site, which usually happens right before you need to predict. So what's surprising, at least to me, is that these early sites here exist, which indicates that the model is aware of what it kind of could say with respect to the space needle much earlier than you would think. After just consuming the subject, it doesn't know yet that I'm looking for a location that is in downtown something, yet it already has a lot of information about the location of the space needle that is associated with the output of Seattle. So let's actually look at what the authors say about these things. I think one component of it is that causal interventions have been shown to be pretty effective at kind of determining what happens in a model. And it seems intuitive, because correlative studies are always kind of – there's always problems with confounding and all things like that. But when we go in and we make explicit changes to the computation of the model and we see what happens, we measure the effects, the things that we can read out are a little bit more clean. So the thing that we do in causal tracing is that the fundamental question is we want to know which of these hidden states is carrying information that can help us convey the factual statement. And like you said, it's a big distributed network. So a priority, one of the things you might think is, well, everything is important and all the states have information that could recover the hidden state. So we wanted to test that. Let's see if this is actually true. So procedurally what causal tracing does is it essentially first obfuscates the subject. It adds noise to the embeddings of the space needle. So now the network doesn't know what you're talking about, and it's got a whole set of corrupted activations. And then the question is, well, if you had clean states, if you could restore any clean state, could you pick one so that after you restored it, the network kind of recoups its computation and that state contains enough information for the rest of the network to determine that the correct answer is Seattle? And so the surprising result is shown in figure 1's E, F, and G, where we see this really, really sharp localization in this specific example. We see a patch that's early and a patch that's late that have really high causal effect. In essence, they have the information that's required to restore the factual statement, but all the other states don't. So a very sparse set of activations that can actually do this. And so we're curious, what does this actually correspond to? So we can actually do this activation copying for specifically the MLP and specifically the attention as well. And what we find is that the MLP corresponds to the early site, and then the attention corresponds to the late site. So the thing is the late site is interesting because, well, it's not exactly too surprising because the model is going to recall the next fact by outputting the next token, so it's right next to the prediction and the causal impact there isn't too surprising. But what's really interesting is this weird early site that seems at first to be in the middle of nowhere. But actually when we do this kind of experiment averaged over a thousand facts, I think that might be figure two or figure... Yeah, it might be on the next page. Yeah. So in figure two, when we do this averaging over a thousand prompts, we find that it systematically lands at the last subject token, this patch of high causal effect in MLPs. And kind of inspired by a lot of the previous work in this area of interpreting where, or in what transformer components are doing, for example, from GAVA, from DAI, and from Alhagi, we sort of form the main hypothesis of the paper that these MLPs are actually what are recalling the factual knowledge. And this is sort of consistent with the transformer circuit's idea that, in particular, Anthropic has been working on, which is that these MLPs might be outputting some kind of information that the attentions that are at the very last token that are actually responsible for the next token prediction are reading. So this was a really stunning surprise to find this kind of separation in such a large network. And the thing that's sort of lucky about it is that MLPs have this really simple form. A lot of work has been done on studying how attention works in these transformers, and attention, my gosh, attention is really complicated. But the MLP, these feedforward layers, they're actually really simple, so they're a pretty interesting thing to study if they're having some decisive effect. So that brought us to the next thing that we did. So just to make it clear, for now, the hypothesis would be something like the MLPs provide information, like they provide some kind of inputs to facts, and then the attention at the later layers will gather all of that information in order to make the final prediction. Yeah, sort of. I think that it's more like, you know, the hypothesis is that the MLPs may be storing this factual knowledge, these factual associations. There's nothing inherent in the words space needle, where you could look at the literal words where it would make sense to predict Seattle. There's a separate association, a separate piece of knowledge that the model has to store somewhere. And the theory is that the association between that word space needle and the location of Seattle is specifically stored in these MLP layers in the middle of the network. So this experiment here is pretty interesting. As far as the way I understand it is the following. The top one, the top is sort of the baseline corrupted input condition. So that baseline corrupted input condition is what we had before as the what happens if we corrupt here the subject. Now not all tokens are shown, but needle is the subject was like space needle was the subject and we corrupt it and we let it run through the network. Now in the original experiment, what we would do is we would copy over from the clean input one of the hidden states, for example, this one right here. However, now we do something in addition. So on the bottom you can see right here, we still do import the clean input right here, as you can see, but then also we take the signals of some of the layers from that corrupted path and we attach them here. Now it sort of takes a moment to kind of estimate what's really happening right here. So it's very interesting to see. Now we measure the causal effect of the of of that node right here as we did before. And here you can see the results as we measure the causal effect. So here effect of a single state, the causal effect is as we discussed before, there is kind of a spike at this early site. However, if we sever the attention modules, we get almost the same effect as you can see right here. Severing is the process I described over to the left right here. However, as we sever the MLP modules, you can see that there is a definite suppression of that effect early on. So where that effect is biggest here originally, it's depressed way down if we sever these MLP connections. So as soon as we import the MLP connections or states, I'd rather want to say the modules, the MLP modules, remember here we're talking about forward signals, not weights. So as soon as we import these for these signals from the MLP modules right here, then we sort of regress back and this node here has no longer much of a causal effect. And that is an indication that the MLP modules might play a major role here in these factual associations. And so what we were asking is, hey, if the MLP modules are so important, what happens if we don't let them read their input? What if we just stuck their input in the fixed corrupted state? So that's what this shortcut is showing these MLP modules, instead of instead of being able to respond to any new information that we're sticking in to clean up the prediction, what if we said the MLP modules aren't allowed to participate in that? So when you do that, normally you have this really strong causal effect for every state that you can see in the purple bars in the graph on the right. But then if you take the MLPs out of the picture, then it drops down to the green bars way below that. So somehow the MLPs at these early layers from about 10 to 20 are really important for this computation. If you take them out, then the causal effects go away. Now, the interesting thing is if you knock out attention the same way, it doesn't really drop that much. So attention is playing some role, but it's not the same important role that MLP is playing. I love this type of research just because on a meta level, it is also really nice to see that research labs, let's say academic labs, can work with... I mean, GPT-2 isn't nowadays one of the largest models in existence, but still it's not all money and compute and scaling up. And you can only get a paper published if you train and train and train and invest. You can do fairly simple things as long as they're smart. And you can find out so much about these things. So I think your paper is also on a meta level, a really good example of what you can still contribute to research even in absence of like giant budgets. I don't know if you have giant budgets, but the paper is certainly doable without, right? If anybody wants to help us with giant budget, then we're always happy to have a little bit more. But the huge models really are doing some really fascinating things. And so we're trying to investigate the really huge models. But yeah, I think that our secret sauce is not compute, our secret sauce is clever experimental design. Yeah. And that it really shows like and the effects here are pretty significant, right? If you cut essentially the contribution of the MLPs, you can see this quite a big drop in the in the causal effect. And it makes it fairly good case, I would say of localizing that knowledge. So now we get to how we kind of determined our hypothesis is not right now that this knowledge, the facts are essentially stored in the MLPs. And if I understand you correctly, something like the space needle is in downtown Seattle, that fact would already be stored in an MLP. And it would be already associated at the point where so here we see at the last subject token, essentially, once I process the space needle, at that point, or maybe one after that, I would have a layer with an MLP in it. And the fact of it being in Seattle would already be stored and recalled at that point to understand you correctly. Yeah. Even though the even though the model doesn't know yet that I'm going to ask it where the space needle is. So that means that essentially, if this hypothesis is correct, the model, once it sees a subject, whatever that means, it will retrieve kind of a whole bunch of knowledge from its different MLPs that are around about the subject for then later, let's say the the attention modules later to aggregate and to retrieve the correct ones from. Yeah, exactly. Right. Yeah. Okay, that's kind of what we found. I think another intuitive hypothesis would also have been that the relation is also encoded in there somewhere. But the challenge there is that the relation often doesn't show up until the very end of the computation. And if you think about it, it's a little bit difficult for facts to be recalled at the very end, because there has to be some kind of general pool of information that you can draw from about a certain subject, even before the question is asked. Yeah. Okay. So MLPs act as key value stores. You want to tell me a little bit about how? Yeah. So this is inspired in part just because of the really nice structure of the MLP simply as two matrices that are connected by a few nonlinearities. But it also draws from research that's been done by GaVa and Dai in the past about a year or two. And basically what they said was that the second MLP or within the MLP, there are two matrices. There's the fan out matrix that gives you a pretty large key space. And then there's a fan back in a matrix that brings it back to the hidden dimension. And so what GaVa found was that the second feed-forward layer seems to act like a key value memory. And they found that a lot of the keys corresponded to a real-life concept. The values, they've shown that sometimes they can correspond to specific embedding vectors. They can correspond, again, to human-identifiable concepts. And so that's one of the things that got us thinking that it was an associative store. But the next thing is simply just that it's a nice matrix. And these matrices have been studied for a long time as methods of storing associations. Like in the very naive case, if you just stuck a fact in every single one of the dimensions, then you would have just n facts that could be stored orthogonally. But there's this really nice interpretation that linear associative memories can store more than the number of rows or columns, depending how you look at it, which is that they minimize squared error between all the key value pairs. And so that sort of gets us started on thinking about how we can take all the associations that are already encoded in this hypothetical matrix and assigning a new association to be constrained as well. The old name for this is linear associated memory. It goes way back to the 1970s, when people were like, what can you use a single layer neural network for? And researchers in the 1970s thought of a lot of alternatives. But one of the leading hypothesis was it just stores key value associations. And they looked at it like a linear least squares problem, that basically you could pack a lot of associations, a lot of remembered values into this key value store. And there might be some error, but a good solution to it would like minimize the squared error. It sort of reduces it to this classical, but actually, you know, pretty straightforward to solve a linear algebra problem. And so that's the old view of it. So now we ask the question, how can we modify such a network such that it kind of learns a new fact or changes its mind about one of the facts that it knows? Well, that in the attack, the attack surface right here is going to be these MLP modules, namely updating the weights of the MLP modules such that they change their mind about a fact. What we would like to do is we have the hypothesis now based on some experiments that the key right here probably corresponds to something like the subject, the space needle, and the value that we get out probably corresponds to something, not exactly the output itself, but kind of that because at that point, it doesn't know yet that I'm looking for a location, right, but probably something like a like a fact about that subject. So I made the example location equals Seattle. So that entire thing, that entire fact could be encoded in this value vector, such that later once it becomes actually clear that I'm looking for a location, that fact can be retrieved as opposed to any of the other facts that would be, let's say stored in any of the other MLPs that the signal is also going through. After all, we're doing multi headed attention. And that's by itself quite an interesting question to ask, like how many facts are there and so on. But I don't want to go into that. The question is, can we change this to something to say location equals Paris? And they go about this fairly in a fairly smart way. And we come back to that at the end or towards the end of the interview, how exactly they they do this. So there's two parts to it. First of all, let's say we know what the key is for the subject. And we know what the value that we'd like to insert is in vector form, like we know the value of this thing. Then they compute, they go through a bit of math here and set this up as a constrained optimization problem. And it turns out if you solve that, then you get a closed form, you get a closed form solution for a rank one update. So they get a closed form solution. That here for and it takes a rank one update that they can easily compute that they need to add to the original weight matrix. And then they essentially get out a updated weight matrix that respects that new fact that they want to insert. And that's what they do. Now, the question is, obviously, how do they know what the vector for the key and the vector for the value is that they want to insert the key is still relatively simple. Since the key is a subject that you know, and want, you can simply let that run through the network and kind of grab the activations at a particular site, they always choose the same site here. But the value is is kind of different. And there, they solve like an optimization problem. So they essentially put the output right here. And I believe in much the same way as like an adversarial example, they they now back optimize what the vector here would need to be in order for the output to change to Paris. This back propagation, this optimization isn't the changing of the network itself, it's simply to compute this V vector right here, so that then then they know how they need to compute the update for the weight matrices. Let's assume that I edit, I say, okay, this is my space needle. And here, I would say no, it's actually in Paris or Rome, not in downtown Seattle. So I want to encode a different value, you phrase this as a constrained minimization problem where I say I want to find a new matrix that still minimizes keys and values, but also obeys my new relation. And you can phrase this then as a closed form, closed form solution. My question is, why did you choose to go with constrained minimization? In this case, why didn't you just ask, add the key here and the value here to all the other keys and values that might already be there, and then essentially minimize the entire thing at once? So one of the reasons is that, so this is a sort of mathematical formulation, but we don't actually have access to all the old keys and values. And so it turns out that if you set it up in the right way, then you can get all the old keys and values to cancel out, so you don't need to know them. And one of the ways to do that is just to set it up as this constrained minimization. The other nice advantage of it is that if you balance this against all the old things, then there's this sort of hyperparameter that you might need to set of how much balance there is. But if we're just setting up a single new fact to learn, it's easiest to just say, you know what? The new model should just know this fact. Let's just know this 100%. And we might have to sacrifice a little bit of increased error on old facts, but there's so many other dimensions that that's just a little bit of error. So we just set it up this way in this paper. Although, setting up the other way that you suggest is a really good idea, and it's actually an approach that we explore in a future paper that hasn't been published yet. But it'll be on archive soon. And hopefully, it's going to be published by the time that this video is released. And I'll point people to it. But essentially, in a nutshell, here, we implant like single new facts into these models. And that works until a couple of dozen facts, maybe. But with your new method, you can implant thousands or even tens of thousands of facts at the same time into networks. Yeah, that's right. Right. So you can really scale this up if you just a few things. If I think about implanting new facts into a network, I can make it really easy for myself. I can just say, you know, whatever, it just needs to fulfill this thing. You know, but I obviously there's a trade off. There's always a trade off, right? Specifically the trade off here is going to be, well, what happens to the rest of the network? Is it still correct? If I tell the network, look, the space needle is actually in Paris, right? What effect does that have on the rest of what the network knows, how it performs and so on? And that's where we get to your fairly extensive, I want to say, evaluation of these things. So we now have an idea of where the facts are. We now have a method to exploit that in order to change those facts. And now what we would love to see is that essentially, well, you tell me what is the ideal outcome of such a method? That's a really interesting question because we spent a lot of time thinking about what should go into counter fact and how to design it so that it's easy to evaluate computationally and stuff like that. But one of the main questions is sort of what does it actually mean to know something, right? What does it mean to have a fact that's actually stored there? And if we think about it, knowledge has, I think, two important properties. Number one, it generalizes. When you rephrase the question, it should be consistent. If you ask a related question that implicitly requires knowledge of that fact, it should also be consistent and all of those things. But at the same time, you can't do this for every single subject in the model. You can't always output Rome or always Paris, always output those kinds of things. So we also want it to be specific. So those are the main two axes on which we measure the edit. Yeah, like what do you mean by specific? Specific as in entities that aren't related, like subjects that aren't related to the subject should not change, essentially. Yeah. So like you move the space needle to Paris, then we don't want to move the Statue of Liberty to Paris at the same time or the Louvre should stay in Paris. What else? What else is in Seattle? Pike's Place. Pike's Place, Mark, shouldn't move to Paris along with the space needle. It should just move one thing. And so the interesting thing is that there does seem to be this tradeoff between being really specific about making a change and having the change be general. And if you sort of change a model without paying too much attention to exactly what you're doing, it's really easy to change a model in a way that is completely generalized but not specific at all. Like everything moves to Paris or vice versa, where it's extremely specific but not generalized at all, where you have a very specific wording of a sentence where now it predicts Paris. But if you change any little detail, then it has no idea what you're talking about. Before you said, OK, we can edit these models and so on, but there are differences and these are the things that you compare with in your evaluation. So you have one evaluation is this zero shot relation extraction, but as I understand it, is not exactly made for your use case. And we need to go further. So you also provide a new data set. Yeah. So a zero shot relation extraction is cool because a lot of previous works in model editing have used it as a baseline. And it actually is quite good. Like you have a bunch of facts you can rewrite. We can paraphrase them. I believe that the ones that we have in our ZSRE data set are the ones that previous works have used are back translated. So we have a few paraphrases. And then we sample a random fact from, I guess, the other facts and check that it changes. So as we can see in the results, there is resolution to the method. We can see various differences in paraphrase and drawdown. But actually, the resolution isn't too high, especially in drawdown. It's hard for any of the really randomly sampled facts to be messed up, even by models that make quite large changes. And also moreover, there's no evaluation of fluency. It's one thing to measure the next token probabilities, but it's also another question of how do we ruin the fluency of the model? Have we deleted so much syntactical knowledge that GPT doesn't generate actual fluent text anymore? So those are a few of the questions that motivate the design of counterfact, which we talk about in the next section. So counterfact is based on something that's very similar to ZSRE. It's actually called Parallel. It's a bunch of relations that some researchers use to analyze how consistent language models are. And basically, it's just a bunch of facts. They're all in the form subject, relation, object. And what we do is we want to test how well the model can be taught facts that aren't already true, because sometimes if you teach it something that it already knows, we might inflate the numbers. So we actually take the objects in all of Parallel and we swap them around. We make everything not true. And then we design a few other things that can help us capture generalization and specificity. Generalization works very similarly to how ZSRE works, where we just paraphrase a bunch of stuff. But specificity is a little bit different, because we found that because of the way that the math works, because we're setting the output of one key to a specific value, if any other keys are in the vicinity of the key that we input or that we edited into the memory, those are pretty vulnerable to moving around. And so what we did for specificity was we looked for neighboring entities that are somewhat related to the subject. And specifically, they're related to the subject because they have a common predicate or the exact same predicate. So if I have the Eiffel Tower and we move it to Rome, then I will look for other things that used to be in Paris, like the Louvre or the Champs-Elysees, things like that. And so that's one of the differences that specificity uses. There's also this fluency and consistency thing, which both deal with generation metrics. So fluency is pretty straightforward. We make it generate some text and we want to see if it's fluent. But then with consistency, we just let the model say whatever it wants about the subject. And we want to see if the keywords that it's outputting actually make sense. For example, if I change the Eiffel Tower to be in Rome, I probably shouldn't see a lot of French vocabulary. I shouldn't see a lot about the food that's in France or the attractions that are in Paris. Or if I move a basketball player to being a football player, he shouldn't be winning the NBA championship. He should be winning the NFL championship or something like that. And so that's another thing that we do. But our hope is that we've designed counter facts so that when you look at all of these five things together, you get a bit of a more complete picture as to what happens to your model after you perform some kind of change. You've talked a bit about generating this data set, seeing, you know, does something make sense and so on. Now we talked about budget before. Is it fair to assume that this data set has at least in part been also generated with the help of automated things like models, or is being also evaluated with the help of automated heuristics? Ah, yeah. Okay. So this data set was actually generated completely computationally. And that's one of the big things with evaluating language, right? It's very hard to design computational metrics that align with human judgment is the short thing. So we actually include a human evaluation. I don't know if we've archived it yet. Yeah, there'll be a human evaluation. But we wanted to balance a few things. But the really nice thing about having things computationally generated is it's very easy to scale it up. So I think one of the secrets and the tricks behind a lot of this knowledge-based work is it actually builds on top of big knowledge graphs and big knowledge bases that have been curated by a lot of people every time. So I think the underlying data underneath parallel and underneath is actually wiki data. And so yeah, how do we get this huge store of predicates to scramble and, you know, related entities to test? They basically come from wiki data. And so that's where we can get the scale for this kind of thing. So down here, you have an example of just one of the edits that you make into the model. So we're dealing with a GPT-2 model right here. And what do we see? What is this here? What is the original fact that the model outputs? Yep, that's correct. And then you decide, no, actually Pierre Curie's area of work is medicine. Now, we haven't talked about yet. Let's go through this step by step. Maybe that's a joke in today's work world. But we're a one-step method. So how would we go about this, because we haven't talked about a final piece of the puzzle yet. We talked about once we have a key and value vector, how do we insert it into an MLP? How do we edit it? But essentially, this now here somehow has to be made into some sort of key and some sort of value. So how do we get these things? Yeah, that's a great question. So the key is a little bit more straightforward, because the natural interpretation of the memory is that once it sees a key, it'll always output a value. And even if it's in the neighborhood, it'll probably output a similar value. So what we can do is we can simply show the model, the subject, and it'll do its computations. And we can collect the activation right before it goes in to the MLP that we're targeting. And that's simply our key. If we want to average across contexts, we can append some text before the subject so that it gets to see what happens to the key when I have five words in front of the subject or 10 words or something like that. And usually it doesn't change too much, but it helps with generalization. But then the value is a little bit more involved. And this is actually an interesting area for future research, because there are a few things and there are lots of things that you could imagine V could be. Like in the most simple, clean case, we would hope that maybe V corresponds to an embedding, for example. So if we want to increase the signal for medicine, we could just add the embedding for medicine or some transformation of the embedding. But as you pointed out earlier, it's not quite that simple, because there are a lot of things that are being stored for Curie. And one of them is that he works in physics or medicine. But also you need to know that he was living in a certain country, he was born in a certain time period, he had friends, x, y, and z, all these kinds of things. So the embedding thing is a little bit simplistic, but it's a super nice ideal to chase. And I think that's an interesting direction of future research. Basically what we do is we perform a little optimization. It's a very constrained optimization, because it's operating only on one vector. Basically what we say is, so the MLP outputs some kind of value. We know that this value is causally important because of the causal tracing stuff. So the question is, how can we tweak this vector so that the new fact is represented instead of the old fact? So we can perform a little optimization. We can say, given that the model currently thinks the answer is Eiffel Towers located in Paris, let's optimize it so that it wants to say Rome instead. And we don't optimize any weights, we don't optimize a huge matrix, we optimize this one little vector that comes out of the MLP. And just changing that vector will allow us to change the final prediction. And in this sense, the optimization takes into account the relation as well, because the backpropagation goes through all the tokens that describe the relation. And so that's sort of what we do. That gives us a vector that'll represent the new fact. Do you want to talk about the tricky second term that you have here? Yeah, sure. So this is, again, indicative of an interesting future research question. But one of the things that we observed, and this is sort of like a limitation, it's an interesting limitation, is that it's very hard to catalog all the things that come out about the subject when you feed the key into the MLP. So there could be a lot of things. And what we've observed is that sometimes we'll observe, we'll see this thing called Essence Drift, which is basically some of the old properties about the subject will change when we didn't want them to change. Like an example of this is, say, you wanted to change Mario Kart to a Microsoft product. If you make the update too strong, it'll actually think Mario Kart is no longer a game, it'll think it's a Microsoft Office productivity tool. And so this last term right here is just to encourage it to not do that. It's basically saying there's some probability distribution over what this subject is, like the essence of the subject, and we want to keep it consistent up to a weighting factor. So admittedly, it's a little bit of a hack, but I think it's useful and it raises this interesting question of how can we decode the vector, the V space as well. And it's simple in the end. I think it takes a few seconds to figure out one of these vectors, and then you can directly write it into the network. It's important to see that these things here, choosing the K vector and ultimately choosing the V vector, are only to figure out the vectors that you then want to put into the network. This optimization procedure doesn't actually change anything in the network. But it's interesting because before you said, essentially, well, we're worried about the keys because keys in the vicinity are subject to change. But now it also turns out that actually values in the vicinity are also subject to change. So if I change the value of a given subject, I need to tell the model, by the way, the rest of the subject is kind of unchanged. Right? Yeah, it's really counterintuitive, right? We have these 1600, 2000 dimensional vector spaces. And I feel like our intuition sometimes fails us. These vector spaces are so big, you really have to respect that you can store a lot of information in just a single vector. Yes, which is so my last question of this would be how do you choose the MLP? Because here you need to target like a specific MLP at a specific layer in the network. How do you choose where you want to make that edit? Yeah. So this is yet another interesting question that kind of foreshadows some of the work that we do in our next paper. But causal tracing gives us sort of a range of MLPs at which it works. And kind of the observation with Rome is that we wanted to make things as simple as possible. And it's fascinating that it works. And possibly a plausible reason for this simplicity is that there's the residual stream, that all these MLPs are contributing towards the hidden state in an additive fashion. So within the band of MLPs that we see high causal effect for, it's plausible that this fact could be stored in any of them. And if any one of them kind of overrides the previous ones, then we'll get the new fact being expressed. And so specifically what we do is we just go to the causal traces and we see where the causal effect peaks. And then we run an experiment that shows that this corresponds pretty well to where the best edit occurs. But basically it's interesting because when you start adding more facts and you need more capacity, the question becomes, well, how do we spread facts across layers? So, you know, what we do is really so, but like, so in a word what we do is really simple. And actually, reviewers didn't really like this as much, right? In GPT-2 XL, we use layer 17, right? We do this causal trace analysis and we find that the causal effects peak there. And we just say, you know, we have all these thousands of facts that we're testing on. We'll just test how well they all can be stored in this specific single matrix at layer 17. And it works pretty darn well. And really, I think it sort of surprised reviewers. They're like, really? Are you, is this all you're doing? But I think the lesson is, if you really map out the mechanisms inside the network, you can get a sense for where things are getting done and you can find the specific location that's most decisive. Now, you're about to talk about scaling. And so I think that if you're trying to insert lots of facts and maybe trying to pile them all into the same matrix, might not scale that well. But for this test that we're doing for this paper, for asking how well can a network absorb a single new written fact, we found that the exact layer that you use may not be so important. If we just picked the single layer that's most effective, then it works for all these facts. So we end up in a situation where we started off by thinking, well, we have this distributed network distributed representations, then you come in and say, no, actually, things are fairly localized, right? They are not only fairly localized, but actually surprisingly, for example, the fact that the space needle might be in Seattle might already be present after the model has consumed space needle as a subject, right, which is fairly surprising. Yeah, now we almost like go a half step back and say, but within that band within sort of that localized area, still, it might be the case that these facts are at least a little bit distributed, right over maybe a bunch of layers adding to the residual stream, which also it's also fascinating that you're saying, well, if I edit if I edit some game to now be a Microsoft game, then all of a sudden, it might think, you know, it's a Microsoft office product or something like this. It's Super Mario is no longer a game, which kind of means that sort of these this this these fact things here, they are not so clean, they are still kind of in super positions with each other, right? If I if I change one, then the others also change a little bit. So I think I think I think the jury is still out. Yeah, like what the structure of that vector space is. And you know, I think there's a difference between knowing whether information is really entangled in that representation, or, or maybe we just haven't developed the right lens or the right method for disentangling the information that's in there. I've seen, I think this morning, I've seen a statistic essentially, listing that as you scale up models, most of the flops, let's say in training and in inference, actually go into the feed forward layers into the MLPs, and not necessarily into the attention mechanisms, everyone's always trying to make attention more efficient, while not realizing that if you really go to these big models, they work in very high vector spaces, and the feed forward layer in a high vector space is actually really, really expensive. Do you think that that fact that we operate in essentially large dimensions and so on that these feed forward layers are so big? Do you think that might be a main contributor to these models essentially performing really well and knowing a lot of things? It would make sense given what you found. I think so. I think these fan out, fan in, feed forward layers are really sponges for information. They can absorb a huge amount of basically memorized information. And so some of that information, you know, our paper is showing some of that information is memorized factual associations. But I think there's a lot of other information that's probably in these matrices as well, you know, information about grammar and lower level things. And so I think that, you know, they're an amazing data structure for knowing a lot. Some of the newer transformers, they add some gating to these MLP layers to, you know, increase their capacity even further. And so I do think it's, they're sort of one of the unsung heroes of these big transformer networks, these huge, massive high capacity memories. Last question from my side. Do you, there's a lot of discussion always about what do these models understand? Now understand is a weak word, a wishy washy word, let's say. But what is your impression? It seems that they certainly do more than just statistical association of kind of tokens to each other. Like what's your current understanding of what are the real understanding capabilities of these models? Do you want to answer that? Do you want me to say something here? It's a loaded question. Yeah, it's a very loaded question. When I like, if we answer this question, then somebody is going to boo us. So I think that, so here's what it seems like to me. There's like positive surprises and some negative surprises. And so, so on the positive side, it was really, really surprising to see that a rank one update in a single layer in a matrix roughly corresponds to what a human thinks of as a fact. Like there's no particular reason that resolution should match so well, right? It could be that a little rank one change in a matrix is much smaller than what a human thinks of as a factor, it could be much bigger, but it actually is kind of surprising that it pretty much matches up pretty well. And so that's really interesting and it raises a bunch of philosophical questions about, you know, what is the nature of knowledge? What is the nature of, you know, the emergence of ideas and big neural networks and so on. But it's pretty cool. On the negative side, there's funny things about the mechanisms that don't really correspond to the way that people think. So I think that the simplest example is like if you reverse the statement of a fact, then these transformers, they process it differently. So for example, if you said Bill Gates, Bill Gates is like Bill Gates is the CEO of Microsoft or founder or maybe. Bill Gates was a founder of Microsoft, right? He's not CEO anymore, he's retired. So but if you said, you know, for example, like if you said Bill Gates was the founder of Microsoft, then you could find that association somewhere in the network. But if you had the network know that, it doesn't necessarily also know that the founder of Microsoft is Bill Gates, because now you've used the other entity as the key and that would that would be potentially stored separately. So if you edited one of those facts, then the other fact wouldn't automatically be edited. You might need a second edit. And and so, you know, that's a little counterintuitive. I think that, you know, if you asked a person, is that one fact that's, oh, yeah, it's a symmetric fact. You know, if you told me one of those, I would know the other. But for a transformer, this may not be the case. It's maybe two separate facts. And that might be I mean, it might be a property of the sort of causal masking that we're doing, right? Because only be able to sort of look back into the sentence already means that you have to pre compute a lot of this knowledge upon seeing the subject. Right. And that might be different paths through the network for the different subjects. So for one subject is Bill Gates and for the other one subject is Microsoft, you don't know what's coming at the end of the sentence. And therefore, you need to be kind of prepared for everything. So maybe bidirectional models might have this differently. Maybe maybe or you could imagine it the other way, because you could also imagine, well, people are constrained to live forward in time. So the way we must think about language must also be, you know, sort of true. So so you have this debate about what is what is the best way to think about it. And and so so so yeah, there's that there's that movie Arrival. I sort of imagined that maybe all the arrival aliens, you know, they they sort of had bidirectional transformer, you know, brains for their language model and and us humans were stuck with these, you know, what you know, unidirectional GPT style models and and that's really hard to communicate between them. Okay, cool. Kevin and David, it was a it was a real pleasure having you here. As I said, I'll link the new paper for sure. And yeah, do you have any last things that you want to get out there to people maybe? How can they get into this field of of knowledge editing and figuring out what these things know? What I what I don't understand. So here's my here's my, you know, question for the machine learning community out there. What I don't understand is why why isn't our entire field about cracking open these models and looking at what's inside them? I think that we're getting better and better at getting really interesting capabilities out of the models, but they contain so many mysteries in there. If you think about the number of billions of parameters inside GPT three, you know, that just like this machine learned code is, you know, it's larger than the entire code base of massive companies that have employed tens of thousands of people to produce, you know, manually produce code for many years. You know, these these large models, they must contain a lot of interesting structure. So so I guess my you know, my my advice is, you know, crack open models. There's there's surely a lot of interesting stuff to discover inside them. Awesome. Kevin last words. Yeah, no, I think this field is very exciting, not only for the I think the science is amazing, but I also think it's it's cool because it inspires interesting questions about what we can do to make these things better. Like some of the negative surprises that we found with, you know, trying to see if GPT really understands certain concepts is that, you know, the observation that there's this bidirectionality of knowledge could only have emerged once we developed a method to edit things to see how work. So I think it's really cool that this kind of stuff can can be raised by interpretability research and it'll help us build better, safer models in the long run that we can actually engineer and I think that's really exciting. All right, cool. Well, thanks so much for being here and best of best of not luck, best of success for the for the future papers. Thanks Yannick. Thank you. It's really nice of you to interview us and it's really great to meet you here. Thank you.
 Hello, today we're talking about locating and editing factual associations in GPT by
 Kevin Meng, David Bao, Alex Andonian and Yonatan Belenkov.
 In this paper, the authors attempt to localize where in a forward pass through a language
 model an actual fact is located or where it is realized.
 For example, something like the Space Needle is in downtown Seattle.
 It has a subject, a verb and an object.
 And where exactly in a language model does the language model know, quote unquote, these
 things and that the Space Needle is in downtown Seattle?
 That's the question of this paper.
 And they go beyond that by figuring out where these facts are.
 They can also then edit those facts, meaning they can change the model such that it all
 of a sudden believes that the Space Needle is in Paris.
 And they test in various ways that this change is first of all robust, it generalizes, but
 it doesn't distort the rest of the model too much.
 Moreover, this change is like a rank one update that they can pre compute.
 So all of this is very, very interesting.
 And we're going into it in detail.
 This video is a bit of a mix between me explaining the paper and the authors with whom I interviewed,
 giving their inputs into various aspects of these questions.
 I hope this is of benefit to you.
 Let me know if you like it or not.
 And let's go into it.
 There's an entire subfield that just researches where are facts in language models.
 I didn't know about the subfield until I read your respective works.
 What does it entail?
 What are people wondering about?
 So I guess there's a few questions.
 I think it's at the intersection of two main things.
 One is a scientific investigation into where things are and what models are doing to achieve
 them.
 And then at the other end of the spectrum is a practical question that sometimes these
 models mess up.
 Because they have information that we want to change because it's now outdated.
 And how do we do this in a practical, in a very clean way?
 On both sides, there are individual respective questions.
 On the interpretability side, I think David might be able to talk about it a bit because
 he's worked with not only language but also vision models.
 But yeah.
 Yeah, so I can talk about the interpretability side.
 Sounds good.
 So on the interpretability side, it's this really old question that's gone back to sort
 of the early days of neuroscience.
 Where do ideas and where does knowledge live in a big neural network?
 People thought about this in the biological neural networks of your brain.
 There's this old theory of the grandmother neuron that maybe you could even have a single
 neuron that's responsible for what you think of your, for thinking about your grandmother.
 Maybe if you pluck that neuron out of your brain, you might forget that whole concept,
 which people think is sort of implausible.
 But what we're chasing here is sort of a weaker locality question.
 Like, if you have some knowledge in a big neural network, can it be localized to a small
 set of neurons or small set of layers?
 Can we find out where that knowledge is?
 And so there's been a bunch of people who have been looking at this.
 It's, you know, I guess maybe the overarching area is called like mechanistic interpretability
 research where people are trying to understand the mechanisms that are emerging inside the
 learned computations.
 And so there's, there was a really nice paper by Al-Haji from, from Anthropic.
 There's been a series of papers from, from JIVA, from, from Israel, who've been looking
 at the structure of computations inside the network.
 And so our paper is another contribution in this direction.
 I think the thing that we're looking at a little differently is we're using, we're
 really focusing on using causal probes to ask that question, you know, making changes
 in the network to see how the network responds when we make changes and using that to map
 out things.
 And what I, what I love about your work is then you actually put it to the test, which
 means that if, if we understand where the knowledge is, we should be able to change
 it, right?
 And that gives to me, the interpretability research is always a bit shrouded in mystery
 because there are always, I feel something like 10,000 different explanations that could
 explain a given fact.
 And usually the researchers frame it in a way that their hypothesis makes the most sense,
 but I'm always like, meh.
 But if you then actually put it to the test and you say, well, if we are correct, we should
 be able to edit the knowledge, we should be able to erase a factor, insert a new one using
 what we think happens.
 And that's also a thing that you do very well.
 Yeah.
 So I think that's where the really interesting interplay between the interpretability and
 the practical side comes in, because on the practical side, people have been chasing this
 question of, of, of, of real world usage.
 Like these models are huge.
 They're really difficult to retrain.
 And then when we actually do fine tune them, for example, on a small data set with a, with
 sort of a blind objective, it's kind of hard to tell sometimes what we're doing with it.
 And so in the past, we've seen some works, for example, from Mitchell and from Decau.
 They spent a lot of time asking the question, like, can we achieve generalization when we
 do edits?
 When we change one thing, does something else change?
 Or is the edit specific?
 Like if we change one thing, does an unrelated fact also change undesirably?
 So they've kind of set this area up because it's a very practical question.
 And I think the really cool thing about Roam is that, like you said, on one side is the
 scientific question, but on the other side, we show that the insights that we get can
 yield a pretty useful model editor that seems to achieve generalization, specificity, and
 fluency preservation all pretty well.
 I was wondering since, since the main foundation of neural networks is distributed representations,
 this is the big step, right, to go from go-fi systems, from symbolic systems to distributed
 systems where we no longer have individual symbols representing individual things in
 the world, which we could build, you know, very simple knowledge graphs.
 Now a fact like the space needle is in downtown Seattle needs to be stored somewhere in a
 vector space.
 Yet you managed to actually locate that fairly well to particular points in the network.
 How does, how does that work?
 So here is how causal tracing works.
 This is one of the main methods the authors employ to figure out where in the model the
 facts are realized.
 We are talking here about the realization of facts, which is connected to the storing
 of facts, but we specifically care about the activation, so the hidden signals as they
 travel through the networks and not necessarily localizing facts inside of the weights of
 the neural network.
 So in this case, you can see that here is a sentence that you input, the space needle
 is in downtown and the model would output, well in this case, it's an uncorrupted sentence,
 the model would get this correct.
 If it's a good language model, you'll get this correct to say Seattle as the next token.
 This as you can see goes through a number of different stages.
 So due to how GPT works, how a autoregressive transformer works with causal masking, you
 will have the word, the token for the being embedded, generating a hidden state here.
 Now that hidden state, first of all, it goes through essentially the layers of the transformers
 and it accumulates two things.
 So it always accumulates an attention head and it accumulates a multi-layer perceptron
 head, or actually, I think two in succession, and then there's a residual connection around
 that.
 So that's what you see right here.
 But also the same hidden signal on each layer travels forward essentially.
 Well, not exactly, it's more like when the second token or the third token, when they
 come in, so when space is now fed into the transformer, it now gets a signal from the
 past, because it does causal attention, it looks at the past.
 So it also will get kind of the hidden signals, the hidden states from the past.
 So essentially this would flow like so, but every time it would also get the hidden signal
 from there.
 And then need will get the hidden signal from both the and space, so it would get both of
 them right here, but also it would travel up the layers and get both the hidden signals
 from here.
 So you can see there is various paths this information can take.
 And the idea here is to figure out where in these hidden states, so in these bubbles right
 here, or this bubble, or this bubble, where is the fact that Seattle should be the output
 of the sentence?
 Where is that kind of realized?
 Where is that localized?
 Now you might have various opinions where that's localized.
 First of all, opinions here, like where in the sentence does the model kind of put a
 lot of weight on Seattle and where in the network?
 So here in the depth of the network, where does that happen?
 And both of them, what turns out as evidence, both of these things are quite surprising.
 So here what they do is this causal tracing.
 What they do is they run the model once with a clean input.
 They record all of these hidden activations.
 Then they run the model again, but this time with corrupted input.
 So here you can see these have little asterisks by them, which means that the input is now
 corrupted.
 It means you add some noise or you just replace them by noise or replace them by something
 else.
 It's just not the original signal anymore.
 And therefore, if you just let the model run, it will probably produce something else because
 the subject, so this is the subject of the sentence, is completely corrupted.
 So this could be whatever is in downtown.
 And then Seattle is certainly not the first thing on the model's mind.
 It might be, but it's like very likely not.
 And then what they do is really interesting.
 They now take each one of these things here individually.
 They take a hidden state and they just copy it over.
 They just copy that over.
 So instead of at this particular hidden state, instead of what the model gets as an input,
 you know, from this path and from this path and from this path here, instead of that,
 it just ignores that particular hidden state and replaces it with the one from the clean
 input.
 And now we observe, so here maybe it said like Paris before because something is in
 downtown, the model just said Paris.
 And now we observe, if it kind of stays at a wrong answer, then that hidden state, that
 original hidden state was probably not super well associated with either the input space
 needle or the output Seattle.
 However, if copying over that hidden state from the clean signal actually changes the
 output back from Paris to Seattle.
 Well, that is a fat marker, oh, sorry about that.
 Those are my notes.
 If that actually changes it back, then we know, aha, this hidden state must be quite
 important for sort of associating space needle to Seattle.
 And that's how we find out.
 And as you can see in the results, you get these two clusters, you get an early, what
 they call an early site, which usually happens after the subject is done, and a late site,
 which usually happens right before you need to predict.
 So what's surprising, at least to me, is that these early sites here exist, which indicates
 that the model is aware of what it kind of could say with respect to the space needle
 much earlier than you would think.
 After just consuming the subject, it doesn't know yet that I'm looking for a location that
 is in downtown something, yet it already has a lot of information about the location of
 the space needle that is associated with the output of Seattle.
 So let's actually look at what the authors say about these things.
 I think one component of it is that causal interventions have been shown to be pretty
 effective at kind of determining what happens in a model.
 And it seems intuitive, because correlative studies are always kind of – there's always
 problems with confounding and all things like that.
 But when we go in and we make explicit changes to the computation of the model and we see
 what happens, we measure the effects, the things that we can read out are a little bit
 more clean.
 So the thing that we do in causal tracing is that the fundamental question is we want
 to know which of these hidden states is carrying information that can help us convey the factual
 statement.
 And like you said, it's a big distributed network.
 So a priority, one of the things you might think is, well, everything is important and
 all the states have information that could recover the hidden state.
 So we wanted to test that.
 Let's see if this is actually true.
 So procedurally what causal tracing does is it essentially first obfuscates the subject.
 It adds noise to the embeddings of the space needle.
 So now the network doesn't know what you're talking about, and it's got a whole set of
 corrupted activations.
 And then the question is, well, if you had clean states, if you could restore any clean
 state, could you pick one so that after you restored it, the network kind of recoups its
 computation and that state contains enough information for the rest of the network to
 determine that the correct answer is Seattle?
 And so the surprising result is shown in figure 1's E, F, and G, where we see this really,
 really sharp localization in this specific example.
 We see a patch that's early and a patch that's late that have really high causal effect.
 In essence, they have the information that's required to restore the factual statement,
 but all the other states don't.
 So a very sparse set of activations that can actually do this.
 And so we're curious, what does this actually correspond to?
 So we can actually do this activation copying for specifically the MLP and specifically
 the attention as well.
 And what we find is that the MLP corresponds to the early site, and then the attention
 corresponds to the late site.
 So the thing is the late site is interesting because, well, it's not exactly too surprising
 because the model is going to recall the next fact by outputting the next token, so it's
 right next to the prediction and the causal impact there isn't too surprising.
 But what's really interesting is this weird early site that seems at first to be in the
 middle of nowhere.
 But actually when we do this kind of experiment averaged over a thousand facts, I think that
 might be figure two or figure...
 Yeah, it might be on the next page.
 Yeah.
 So in figure two, when we do this averaging over a thousand prompts, we find that it systematically
 lands at the last subject token, this patch of high causal effect in MLPs.
 And kind of inspired by a lot of the previous work in this area of interpreting where, or
 in what transformer components are doing, for example, from GAVA, from DAI, and from
 Alhagi, we sort of form the main hypothesis of the paper that these MLPs are actually
 what are recalling the factual knowledge.
 And this is sort of consistent with the transformer circuit's idea that, in particular, Anthropic
 has been working on, which is that these MLPs might be outputting some kind of information
 that the attentions that are at the very last token that are actually responsible for the
 next token prediction are reading.
 So this was a really stunning surprise to find this kind of separation in such a large
 network.
 And the thing that's sort of lucky about it is that MLPs have this really simple form.
 A lot of work has been done on studying how attention works in these transformers, and
 attention, my gosh, attention is really complicated.
 But the MLP, these feedforward layers, they're actually really simple, so they're a pretty
 interesting thing to study if they're having some decisive effect.
 So that brought us to the next thing that we did.
 So just to make it clear, for now, the hypothesis would be something like the MLPs provide information,
 like they provide some kind of inputs to facts, and then the attention at the later layers
 will gather all of that information in order to make the final prediction.
 Yeah, sort of.
 I think that it's more like, you know, the hypothesis is that the MLPs may be storing
 this factual knowledge, these factual associations.
 There's nothing inherent in the words space needle, where you could look at the literal
 words where it would make sense to predict Seattle.
 There's a separate association, a separate piece of knowledge that the model has to store
 somewhere.
 And the theory is that the association between that word space needle and the location of
 Seattle is specifically stored in these MLP layers in the middle of the network.
 So this experiment here is pretty interesting.
 As far as the way I understand it is the following.
 The top one, the top is sort of the baseline corrupted input condition.
 So that baseline corrupted input condition is what we had before as the what happens
 if we corrupt here the subject.
 Now not all tokens are shown, but needle is the subject was like space needle was the
 subject and we corrupt it and we let it run through the network.
 Now in the original experiment, what we would do is we would copy over from the clean input
 one of the hidden states, for example, this one right here.
 However, now we do something in addition.
 So on the bottom you can see right here, we still do import the clean input right here,
 as you can see, but then also we take the signals of some of the layers from that corrupted
 path and we attach them here.
 Now it sort of takes a moment to kind of estimate what's really happening right here.
 So it's very interesting to see.
 Now we measure the causal effect of the of of that node right here as we did before.
 And here you can see the results as we measure the causal effect.
 So here effect of a single state, the causal effect is as we discussed before, there is
 kind of a spike at this early site.
 However, if we sever the attention modules, we get almost the same effect as you can see
 right here.
 Severing is the process I described over to the left right here.
 However, as we sever the MLP modules, you can see that there is a definite suppression
 of that effect early on.
 So where that effect is biggest here originally, it's depressed way down if we sever these
 MLP connections.
 So as soon as we import the MLP connections or states, I'd rather want to say the modules,
 the MLP modules, remember here we're talking about forward signals, not weights.
 So as soon as we import these for these signals from the MLP modules right here, then we sort
 of regress back and this node here has no longer much of a causal effect.
 And that is an indication that the MLP modules might play a major role here in these factual
 associations.
 And so what we were asking is, hey, if the MLP modules are so important, what happens
 if we don't let them read their input?
 What if we just stuck their input in the fixed corrupted state?
 So that's what this shortcut is showing these MLP modules, instead of instead of being able
 to respond to any new information that we're sticking in to clean up the prediction, what
 if we said the MLP modules aren't allowed to participate in that?
 So when you do that, normally you have this really strong causal effect for every state
 that you can see in the purple bars in the graph on the right.
 But then if you take the MLPs out of the picture, then it drops down to the green bars way below
 that.
 So somehow the MLPs at these early layers from about 10 to 20 are really important for
 this computation.
 If you take them out, then the causal effects go away.
 Now, the interesting thing is if you knock out attention the same way, it doesn't really
 drop that much.
 So attention is playing some role, but it's not the same important role that MLP is playing.
 I love this type of research just because on a meta level, it is also really nice to
 see that research labs, let's say academic labs, can work with...
 I mean, GPT-2 isn't nowadays one of the largest models in existence, but still it's not all
 money and compute and scaling up.
 And you can only get a paper published if you train and train and train and invest.
 You can do fairly simple things as long as they're smart.
 And you can find out so much about these things.
 So I think your paper is also on a meta level, a really good example of what you can still
 contribute to research even in absence of like giant budgets.
 I don't know if you have giant budgets, but the paper is certainly doable without, right?
 If anybody wants to help us with giant budget, then we're always happy to have a little bit
 more.
 But the huge models really are doing some really fascinating things.
 And so we're trying to investigate the really huge models.
 But yeah, I think that our secret sauce is not compute, our secret sauce is clever experimental
 design.
 Yeah.
 And that it really shows like and the effects here are pretty significant, right?
 If you cut essentially the contribution of the MLPs, you can see this quite a big drop
 in the in the causal effect.
 And it makes it fairly good case, I would say of localizing that knowledge.
 So now we get to how we kind of determined our hypothesis is not right now that this
 knowledge, the facts are essentially stored in the MLPs.
 And if I understand you correctly, something like the space needle is in downtown Seattle,
 that fact would already be stored in an MLP.
 And it would be already associated at the point where so here we see at the last subject
 token, essentially, once I process the space needle, at that point, or maybe one after
 that, I would have a layer with an MLP in it.
 And the fact of it being in Seattle would already be stored and recalled at that point
 to understand you correctly.
 Yeah.
 Even though the even though the model doesn't know yet that I'm going to ask it where the
 space needle is.
 So that means that essentially, if this hypothesis is correct, the model, once it sees a subject,
 whatever that means, it will retrieve kind of a whole bunch of knowledge from its different
 MLPs that are around about the subject for then later, let's say the the attention modules
 later to aggregate and to retrieve the correct ones from.
 Yeah, exactly.
 Right.
 Yeah.
 Okay, that's kind of what we found.
 I think another intuitive hypothesis would also have been that the relation is also encoded
 in there somewhere.
 But the challenge there is that the relation often doesn't show up until the very end of
 the computation.
 And if you think about it, it's a little bit difficult for facts to be recalled at the
 very end, because there has to be some kind of general pool of information that you can
 draw from about a certain subject, even before the question is asked.
 Yeah.
 Okay.
 So MLPs act as key value stores.
 You want to tell me a little bit about how?
 Yeah.
 So this is inspired in part just because of the really nice structure of the MLP simply
 as two matrices that are connected by a few nonlinearities.
 But it also draws from research that's been done by GaVa and Dai in the past about a year
 or two.
 And basically what they said was that the second MLP or within the MLP, there are two
 matrices.
 There's the fan out matrix that gives you a pretty large key space.
 And then there's a fan back in a matrix that brings it back to the hidden dimension.
 And so what GaVa found was that the second feed-forward layer seems to act like a key
 value memory.
 And they found that a lot of the keys corresponded to a real-life concept.
 The values, they've shown that sometimes they can correspond to specific embedding vectors.
 They can correspond, again, to human-identifiable concepts.
 And so that's one of the things that got us thinking that it was an associative store.
 But the next thing is simply just that it's a nice matrix.
 And these matrices have been studied for a long time as methods of storing associations.
 Like in the very naive case, if you just stuck a fact in every single one of the dimensions,
 then you would have just n facts that could be stored orthogonally.
 But there's this really nice interpretation that linear associative memories can store
 more than the number of rows or columns, depending how you look at it, which is that they minimize
 squared error between all the key value pairs.
 And so that sort of gets us started on thinking about how we can take all the associations
 that are already encoded in this hypothetical matrix and assigning a new association to
 be constrained as well.
 The old name for this is linear associated memory.
 It goes way back to the 1970s, when people were like, what can you use a single layer
 neural network for?
 And researchers in the 1970s thought of a lot of alternatives.
 But one of the leading hypothesis was it just stores key value associations.
 And they looked at it like a linear least squares problem, that basically you could
 pack a lot of associations, a lot of remembered values into this key value store.
 And there might be some error, but a good solution to it would like minimize the squared
 error.
 It sort of reduces it to this classical, but actually, you know, pretty straightforward
 to solve a linear algebra problem.
 And so that's the old view of it.
 So now we ask the question, how can we modify such a network such that it kind of learns
 a new fact or changes its mind about one of the facts that it knows?
 Well, that in the attack, the attack surface right here is going to be these MLP modules,
 namely updating the weights of the MLP modules such that they change their mind about a fact.
 What we would like to do is we have the hypothesis now based on some experiments that the key
 right here probably corresponds to something like the subject, the space needle, and the
 value that we get out probably corresponds to something, not exactly the output itself,
 but kind of that because at that point, it doesn't know yet that I'm looking for a location,
 right, but probably something like a like a fact about that subject.
 So I made the example location equals Seattle.
 So that entire thing, that entire fact could be encoded in this value vector, such that
 later once it becomes actually clear that I'm looking for a location, that fact can
 be retrieved as opposed to any of the other facts that would be, let's say stored in any
 of the other MLPs that the signal is also going through.
 After all, we're doing multi headed attention.
 And that's by itself quite an interesting question to ask, like how many facts are there
 and so on.
 But I don't want to go into that.
 The question is, can we change this to something to say location equals Paris?
 And they go about this fairly in a fairly smart way.
 And we come back to that at the end or towards the end of the interview, how exactly they
 they do this.
 So there's two parts to it.
 First of all, let's say we know what the key is for the subject.
 And we know what the value that we'd like to insert is in vector form, like we know
 the value of this thing.
 Then they compute, they go through a bit of math here and set this up as a constrained
 optimization problem.
 And it turns out if you solve that, then you get a closed form, you get a closed form solution
 for a rank one update.
 So they get a closed form solution.
 That here for and it takes a rank one update that they can easily compute that they need
 to add to the original weight matrix.
 And then they essentially get out a updated weight matrix that respects that new fact
 that they want to insert.
 And that's what they do.
 Now, the question is, obviously, how do they know what the vector for the key and the vector
 for the value is that they want to insert the key is still relatively simple.
 Since the key is a subject that you know, and want, you can simply let that run through
 the network and kind of grab the activations at a particular site, they always choose the
 same site here.
 But the value is is kind of different.
 And there, they solve like an optimization problem.
 So they essentially put the output right here.
 And I believe in much the same way as like an adversarial example, they they now back
 optimize what the vector here would need to be in order for the output to change to Paris.
 This back propagation, this optimization isn't the changing of the network itself, it's simply
 to compute this V vector right here, so that then then they know how they need to compute
 the update for the weight matrices.
 Let's assume that I edit, I say, okay, this is my space needle.
 And here, I would say no, it's actually in Paris or Rome, not in downtown Seattle.
 So I want to encode a different value, you phrase this as a constrained minimization
 problem where I say I want to find a new matrix that still minimizes keys and values, but
 also obeys my new relation.
 And you can phrase this then as a closed form, closed form solution.
 My question is, why did you choose to go with constrained minimization?
 In this case, why didn't you just ask, add the key here and the value here to all the
 other keys and values that might already be there, and then essentially minimize the entire
 thing at once?
 So one of the reasons is that, so this is a sort of mathematical formulation, but we
 don't actually have access to all the old keys and values.
 And so it turns out that if you set it up in the right way, then you can get all the
 old keys and values to cancel out, so you don't need to know them.
 And one of the ways to do that is just to set it up as this constrained minimization.
 The other nice advantage of it is that if you balance this against all the old things,
 then there's this sort of hyperparameter that you might need to set of how much balance
 there is.
 But if we're just setting up a single new fact to learn, it's easiest to just say, you
 know what?
 The new model should just know this fact.
 Let's just know this 100%.
 And we might have to sacrifice a little bit of increased error on old facts, but there's
 so many other dimensions that that's just a little bit of error.
 So we just set it up this way in this paper.
 Although, setting up the other way that you suggest is a really good idea, and it's actually
 an approach that we explore in a future paper that hasn't been published yet.
 But it'll be on archive soon.
 And hopefully, it's going to be published by the time that this video is released.
 And I'll point people to it.
 But essentially, in a nutshell, here, we implant like single new facts into these models.
 And that works until a couple of dozen facts, maybe.
 But with your new method, you can implant thousands or even tens of thousands of facts
 at the same time into networks.
 Yeah, that's right.
 Right.
 So you can really scale this up if you just a few things.
 If I think about implanting new facts into a network, I can make it really easy for myself.
 I can just say, you know, whatever, it just needs to fulfill this thing.
 You know, but I obviously there's a trade off.
 There's always a trade off, right?
 Specifically the trade off here is going to be, well, what happens to the rest of the
 network?
 Is it still correct?
 If I tell the network, look, the space needle is actually in Paris, right?
 What effect does that have on the rest of what the network knows, how it performs and
 so on?
 And that's where we get to your fairly extensive, I want to say, evaluation of these things.
 So we now have an idea of where the facts are.
 We now have a method to exploit that in order to change those facts.
 And now what we would love to see is that essentially, well, you tell me what is the
 ideal outcome of such a method?
 That's a really interesting question because we spent a lot of time thinking about what
 should go into counter fact and how to design it so that it's easy to evaluate computationally
 and stuff like that.
 But one of the main questions is sort of what does it actually mean to know something, right?
 What does it mean to have a fact that's actually stored there?
 And if we think about it, knowledge has, I think, two important properties.
 Number one, it generalizes.
 When you rephrase the question, it should be consistent.
 If you ask a related question that implicitly requires knowledge of that fact, it should
 also be consistent and all of those things.
 But at the same time, you can't do this for every single subject in the model.
 You can't always output Rome or always Paris, always output those kinds of things.
 So we also want it to be specific.
 So those are the main two axes on which we measure the edit.
 Yeah, like what do you mean by specific?
 Specific as in entities that aren't related, like subjects that aren't related to the subject
 should not change, essentially.
 Yeah.
 So like you move the space needle to Paris, then we don't want to move the Statue of Liberty
 to Paris at the same time or the Louvre should stay in Paris.
 What else?
 What else is in Seattle?
 Pike's Place.
 Pike's Place, Mark, shouldn't move to Paris along with the space needle.
 It should just move one thing.
 And so the interesting thing is that there does seem to be this tradeoff between being
 really specific about making a change and having the change be general.
 And if you sort of change a model without paying too much attention to exactly what
 you're doing, it's really easy to change a model in a way that is completely generalized
 but not specific at all.
 Like everything moves to Paris or vice versa, where it's extremely specific but not generalized
 at all, where you have a very specific wording of a sentence where now it predicts Paris.
 But if you change any little detail, then it has no idea what you're talking about.
 Before you said, OK, we can edit these models and so on, but there are differences and these
 are the things that you compare with in your evaluation.
 So you have one evaluation is this zero shot relation extraction, but as I understand it,
 is not exactly made for your use case.
 And we need to go further.
 So you also provide a new data set.
 Yeah.
 So a zero shot relation extraction is cool because a lot of previous works in model editing
 have used it as a baseline.
 And it actually is quite good.
 Like you have a bunch of facts you can rewrite.
 We can paraphrase them.
 I believe that the ones that we have in our ZSRE data set are the ones that previous works
 have used are back translated.
 So we have a few paraphrases.
 And then we sample a random fact from, I guess, the other facts and check that it changes.
 So as we can see in the results, there is resolution to the method.
 We can see various differences in paraphrase and drawdown.
 But actually, the resolution isn't too high, especially in drawdown.
 It's hard for any of the really randomly sampled facts to be messed up, even by models that
 make quite large changes.
 And also moreover, there's no evaluation of fluency.
 It's one thing to measure the next token probabilities, but it's also another question of how do
 we ruin the fluency of the model?
 Have we deleted so much syntactical knowledge that GPT doesn't generate actual fluent text
 anymore?
 So those are a few of the questions that motivate the design of counterfact, which we talk about
 in the next section.
 So counterfact is based on something that's very similar to ZSRE.
 It's actually called Parallel.
 It's a bunch of relations that some researchers use to analyze how consistent language models
 are.
 And basically, it's just a bunch of facts.
 They're all in the form subject, relation, object.
 And what we do is we want to test how well the model can be taught facts that aren't
 already true, because sometimes if you teach it something that it already knows, we might
 inflate the numbers.
 So we actually take the objects in all of Parallel and we swap them around.
 We make everything not true.
 And then we design a few other things that can help us capture generalization and specificity.
 Generalization works very similarly to how ZSRE works, where we just paraphrase a bunch
 of stuff.
 But specificity is a little bit different, because we found that because of the way that
 the math works, because we're setting the output of one key to a specific value, if
 any other keys are in the vicinity of the key that we input or that we edited into the
 memory, those are pretty vulnerable to moving around.
 And so what we did for specificity was we looked for neighboring entities that are somewhat
 related to the subject.
 And specifically, they're related to the subject because they have a common predicate or the
 exact same predicate.
 So if I have the Eiffel Tower and we move it to Rome, then I will look for other things
 that used to be in Paris, like the Louvre or the Champs-Elysees, things like that.
 And so that's one of the differences that specificity uses.
 There's also this fluency and consistency thing, which both deal with generation metrics.
 So fluency is pretty straightforward.
 We make it generate some text and we want to see if it's fluent.
 But then with consistency, we just let the model say whatever it wants about the subject.
 And we want to see if the keywords that it's outputting actually make sense.
 For example, if I change the Eiffel Tower to be in Rome, I probably shouldn't see a
 lot of French vocabulary.
 I shouldn't see a lot about the food that's in France or the attractions that are in Paris.
 Or if I move a basketball player to being a football player, he shouldn't be winning
 the NBA championship.
 He should be winning the NFL championship or something like that.
 And so that's another thing that we do.
 But our hope is that we've designed counter facts so that when you look at all of these
 five things together, you get a bit of a more complete picture as to what happens to your
 model after you perform some kind of change.
 You've talked a bit about generating this data set, seeing, you know, does something
 make sense and so on.
 Now we talked about budget before.
 Is it fair to assume that this data set has at least in part been also generated with
 the help of automated things like models, or is being also evaluated with the help of
 automated heuristics?
 Ah, yeah.
 Okay.
 So this data set was actually generated completely computationally.
 And that's one of the big things with evaluating language, right?
 It's very hard to design computational metrics that align with human judgment is the short
 thing.
 So we actually include a human evaluation.
 I don't know if we've archived it yet.
 Yeah, there'll be a human evaluation.
 But we wanted to balance a few things.
 But the really nice thing about having things computationally generated is it's very easy
 to scale it up.
 So I think one of the secrets and the tricks behind a lot of this knowledge-based work
 is it actually builds on top of big knowledge graphs and big knowledge bases that have been
 curated by a lot of people every time.
 So I think the underlying data underneath parallel and underneath is actually wiki data.
 And so yeah, how do we get this huge store of predicates to scramble and, you know, related
 entities to test?
 They basically come from wiki data.
 And so that's where we can get the scale for this kind of thing.
 So down here, you have an example of just one of the edits that you make into the model.
 So we're dealing with a GPT-2 model right here.
 And what do we see?
 What is this here?
 What is the original fact that the model outputs?
 Yep, that's correct.
 And then you decide, no, actually Pierre Curie's area of work is medicine.
 Now, we haven't talked about yet.
 Let's go through this step by step.
 Maybe that's a joke in today's work world.
 But we're a one-step method.
 So how would we go about this, because we haven't talked about a final piece of the
 puzzle yet.
 We talked about once we have a key and value vector, how do we insert it into an MLP?
 How do we edit it?
 But essentially, this now here somehow has to be made into some sort of key and some
 sort of value.
 So how do we get these things?
 Yeah, that's a great question.
 So the key is a little bit more straightforward, because the natural interpretation of the
 memory is that once it sees a key, it'll always output a value.
 And even if it's in the neighborhood, it'll probably output a similar value.
 So what we can do is we can simply show the model, the subject, and it'll do its computations.
 And we can collect the activation right before it goes in to the MLP that we're targeting.
 And that's simply our key.
 If we want to average across contexts, we can append some text before the subject so
 that it gets to see what happens to the key when I have five words in front of the subject
 or 10 words or something like that.
 And usually it doesn't change too much, but it helps with generalization.
 But then the value is a little bit more involved.
 And this is actually an interesting area for future research, because there are a few things
 and there are lots of things that you could imagine V could be.
 Like in the most simple, clean case, we would hope that maybe V corresponds to an embedding,
 for example.
 So if we want to increase the signal for medicine, we could just add the embedding for medicine
 or some transformation of the embedding.
 But as you pointed out earlier, it's not quite that simple, because there are a lot of things
 that are being stored for Curie.
 And one of them is that he works in physics or medicine.
 But also you need to know that he was living in a certain country, he was born in a certain
 time period, he had friends, x, y, and z, all these kinds of things.
 So the embedding thing is a little bit simplistic, but it's a super nice ideal to chase.
 And I think that's an interesting direction of future research.
 Basically what we do is we perform a little optimization.
 It's a very constrained optimization, because it's operating only on one vector.
 Basically what we say is, so the MLP outputs some kind of value.
 We know that this value is causally important because of the causal tracing stuff.
 So the question is, how can we tweak this vector so that the new fact is represented
 instead of the old fact?
 So we can perform a little optimization.
 We can say, given that the model currently thinks the answer is Eiffel Towers located
 in Paris, let's optimize it so that it wants to say Rome instead.
 And we don't optimize any weights, we don't optimize a huge matrix, we optimize this one
 little vector that comes out of the MLP.
 And just changing that vector will allow us to change the final prediction.
 And in this sense, the optimization takes into account the relation as well, because
 the backpropagation goes through all the tokens that describe the relation.
 And so that's sort of what we do.
 That gives us a vector that'll represent the new fact.
 Do you want to talk about the tricky second term that you have here?
 Yeah, sure.
 So this is, again, indicative of an interesting future research question.
 But one of the things that we observed, and this is sort of like a limitation, it's an
 interesting limitation, is that it's very hard to catalog all the things that come out
 about the subject when you feed the key into the MLP.
 So there could be a lot of things.
 And what we've observed is that sometimes we'll observe, we'll see this thing called
 Essence Drift, which is basically some of the old properties about the subject will
 change when we didn't want them to change.
 Like an example of this is, say, you wanted to change Mario Kart to a Microsoft product.
 If you make the update too strong, it'll actually think Mario Kart is no longer a game, it'll
 think it's a Microsoft Office productivity tool.
 And so this last term right here is just to encourage it to not do that.
 It's basically saying there's some probability distribution over what this subject is, like
 the essence of the subject, and we want to keep it consistent up to a weighting factor.
 So admittedly, it's a little bit of a hack, but I think it's useful and it raises this
 interesting question of how can we decode the vector, the V space as well.
 And it's simple in the end.
 I think it takes a few seconds to figure out one of these vectors, and then you can directly
 write it into the network.
 It's important to see that these things here, choosing the K vector and ultimately choosing
 the V vector, are only to figure out the vectors that you then want to put into the network.
 This optimization procedure doesn't actually change anything in the network.
 But it's interesting because before you said, essentially, well, we're worried about the
 keys because keys in the vicinity are subject to change.
 But now it also turns out that actually values in the vicinity are also subject to change.
 So if I change the value of a given subject, I need to tell the model, by the way, the
 rest of the subject is kind of unchanged.
 Right?
 Yeah, it's really counterintuitive, right?
 We have these 1600, 2000 dimensional vector spaces.
 And I feel like our intuition sometimes fails us.
 These vector spaces are so big, you really have to respect that you can store a lot of
 information in just a single vector.
 Yes, which is so my last question of this would be how do you choose the MLP?
 Because here you need to target like a specific MLP at a specific layer in the network.
 How do you choose where you want to make that edit?
 Yeah.
 So this is yet another interesting question that kind of foreshadows some of the work
 that we do in our next paper.
 But causal tracing gives us sort of a range of MLPs at which it works.
 And kind of the observation with Rome is that we wanted to make things as simple as possible.
 And it's fascinating that it works.
 And possibly a plausible reason for this simplicity is that there's the residual stream, that
 all these MLPs are contributing towards the hidden state in an additive fashion.
 So within the band of MLPs that we see high causal effect for, it's plausible that this
 fact could be stored in any of them.
 And if any one of them kind of overrides the previous ones, then we'll get the new fact
 being expressed.
 And so specifically what we do is we just go to the causal traces and we see where the
 causal effect peaks.
 And then we run an experiment that shows that this corresponds pretty well to where the
 best edit occurs.
 But basically it's interesting because when you start adding more facts and you need more
 capacity, the question becomes, well, how do we spread facts across layers?
 So, you know, what we do is really so, but like, so in a word what we do is really simple.
 And actually, reviewers didn't really like this as much, right?
 In GPT-2 XL, we use layer 17, right?
 We do this causal trace analysis and we find that the causal effects peak there.
 And we just say, you know, we have all these thousands of facts that we're testing on.
 We'll just test how well they all can be stored in this specific single matrix at layer 17.
 And it works pretty darn well.
 And really, I think it sort of surprised reviewers.
 They're like, really?
 Are you, is this all you're doing?
 But I think the lesson is, if you really map out the mechanisms inside the network, you
 can get a sense for where things are getting done and you can find the specific location
 that's most decisive.
 Now, you're about to talk about scaling.
 And so I think that if you're trying to insert lots of facts and maybe trying to pile them
 all into the same matrix, might not scale that well.
 But for this test that we're doing for this paper, for asking how well can a network absorb
 a single new written fact, we found that the exact layer that you use may not be so important.
 If we just picked the single layer that's most effective, then it works for all these
 facts.
 So we end up in a situation where we started off by thinking, well, we have this distributed
 network distributed representations, then you come in and say, no, actually, things
 are fairly localized, right?
 They are not only fairly localized, but actually surprisingly, for example, the fact that the
 space needle might be in Seattle might already be present after the model has consumed space
 needle as a subject, right, which is fairly surprising.
 Yeah, now we almost like go a half step back and say, but within that band within sort
 of that localized area, still, it might be the case that these facts are at least a little
 bit distributed, right over maybe a bunch of layers adding to the residual stream, which
 also it's also fascinating that you're saying, well, if I edit if I edit some game to now
 be a Microsoft game, then all of a sudden, it might think, you know, it's a Microsoft
 office product or something like this.
 It's Super Mario is no longer a game, which kind of means that sort of these this this
 these fact things here, they are not so clean, they are still kind of in super positions
 with each other, right?
 If I if I change one, then the others also change a little bit.
 So I think I think I think the jury is still out.
 Yeah, like what the structure of that vector space is.
 And you know, I think there's a difference between knowing whether information is really
 entangled in that representation, or, or maybe we just haven't developed the right lens or
 the right method for disentangling the information that's in there.
 I've seen, I think this morning, I've seen a statistic essentially, listing that as you
 scale up models, most of the flops, let's say in training and in inference, actually
 go into the feed forward layers into the MLPs, and not necessarily into the attention mechanisms,
 everyone's always trying to make attention more efficient, while not realizing that if
 you really go to these big models, they work in very high vector spaces, and the feed forward
 layer in a high vector space is actually really, really expensive.
 Do you think that that fact that we operate in essentially large dimensions and so on
 that these feed forward layers are so big?
 Do you think that might be a main contributor to these models essentially performing really
 well and knowing a lot of things?
 It would make sense given what you found.
 I think so.
 I think these fan out, fan in, feed forward layers are really sponges for information.
 They can absorb a huge amount of basically memorized information.
 And so some of that information, you know, our paper is showing some of that information
 is memorized factual associations.
 But I think there's a lot of other information that's probably in these matrices as well,
 you know, information about grammar and lower level things.
 And so I think that, you know, they're an amazing data structure for knowing a lot.
 Some of the newer transformers, they add some gating to these MLP layers to, you know, increase
 their capacity even further.
 And so I do think it's, they're sort of one of the unsung heroes of these big transformer
 networks, these huge, massive high capacity memories.
 Last question from my side.
 Do you, there's a lot of discussion always about what do these models understand?
 Now understand is a weak word, a wishy washy word, let's say.
 But what is your impression?
 It seems that they certainly do more than just statistical association of kind of tokens
 to each other.
 Like what's your current understanding of what are the real understanding capabilities
 of these models?
 Do you want to answer that?
 Do you want me to say something here?
 It's a loaded question.
 Yeah, it's a very loaded question.
 When I like, if we answer this question, then somebody is going to boo us.
 So I think that, so here's what it seems like to me.
 There's like positive surprises and some negative surprises.
 And so, so on the positive side, it was really, really surprising to see that a rank one update
 in a single layer in a matrix roughly corresponds to what a human thinks of as a fact.
 Like there's no particular reason that resolution should match so well, right?
 It could be that a little rank one change in a matrix is much smaller than what a human
 thinks of as a factor, it could be much bigger, but it actually is kind of surprising that
 it pretty much matches up pretty well.
 And so that's really interesting and it raises a bunch of philosophical questions about,
 you know, what is the nature of knowledge?
 What is the nature of, you know, the emergence of ideas and big neural networks and so on.
 But it's pretty cool.
 On the negative side, there's funny things about the mechanisms that don't really correspond
 to the way that people think.
 So I think that the simplest example is like if you reverse the statement of a fact, then
 these transformers, they process it differently.
 So for example, if you said Bill Gates, Bill Gates is like Bill Gates is the CEO of Microsoft
 or founder or maybe.
 Bill Gates was a founder of Microsoft, right?
 He's not CEO anymore, he's retired.
 So but if you said, you know, for example, like if you said Bill Gates was the founder
 of Microsoft, then you could find that association somewhere in the network.
 But if you had the network know that, it doesn't necessarily also know that the founder of
 Microsoft is Bill Gates, because now you've used the other entity as the key and that
 would that would be potentially stored separately.
 So if you edited one of those facts, then the other fact wouldn't automatically be edited.
 You might need a second edit.
 And and so, you know, that's a little counterintuitive.
 I think that, you know, if you asked a person, is that one fact that's, oh, yeah, it's a
 symmetric fact.
 You know, if you told me one of those, I would know the other.
 But for a transformer, this may not be the case.
 It's maybe two separate facts.
 And that might be I mean, it might be a property of the sort of causal masking that we're doing,
 right?
 Because only be able to sort of look back into the sentence already means that you have
 to pre compute a lot of this knowledge upon seeing the subject.
 Right.
 And that might be different paths through the network for the different subjects.
 So for one subject is Bill Gates and for the other one subject is Microsoft, you don't
 know what's coming at the end of the sentence.
 And therefore, you need to be kind of prepared for everything.
 So maybe bidirectional models might have this differently.
 Maybe maybe or you could imagine it the other way, because you could also imagine, well,
 people are constrained to live forward in time.
 So the way we must think about language must also be, you know, sort of true.
 So so you have this debate about what is what is the best way to think about it.
 And and so so so yeah, there's that there's that movie Arrival.
 I sort of imagined that maybe all the arrival aliens, you know, they they sort of had bidirectional
 transformer, you know, brains for their language model and and us humans were stuck with these,
 you know, what you know, unidirectional GPT style models and and that's really hard to
 communicate between them.
 Okay, cool.
 Kevin and David, it was a it was a real pleasure having you here.
 As I said, I'll link the new paper for sure.
 And yeah, do you have any last things that you want to get out there to people maybe?
 How can they get into this field of of knowledge editing and figuring out what these things
 know?
 What I what I don't understand.
 So here's my here's my, you know, question for the machine learning community out there.
 What I don't understand is why why isn't our entire field about cracking open these models
 and looking at what's inside them?
 I think that we're getting better and better at getting really interesting capabilities
 out of the models, but they contain so many mysteries in there.
 If you think about the number of billions of parameters inside GPT three, you know, that
 just like this machine learned code is, you know, it's larger than the entire code base
 of massive companies that have employed tens of thousands of people to produce, you know,
 manually produce code for many years.
 You know, these these large models, they must contain a lot of interesting structure.
 So so I guess my you know, my my advice is, you know, crack open models.
 There's there's surely a lot of interesting stuff to discover inside them.
 Awesome.
 Kevin last words.
 Yeah, no, I think this field is very exciting, not only for the I think the science is amazing,
 but I also think it's it's cool because it inspires interesting questions about what
 we can do to make these things better.
 Like some of the negative surprises that we found with, you know, trying to see if GPT
 really understands certain concepts is that, you know, the observation that there's this
 bidirectionality of knowledge could only have emerged once we developed a method to edit
 things to see how work.
 So I think it's really cool that this kind of stuff can can be raised by interpretability
 research and it'll help us build better, safer models in the long run that we can actually
 engineer and I think that's really exciting.
 All right, cool.
 Well, thanks so much for being here and best of best of not luck, best of success for the
 for the future papers.
 Thanks Yannick.
 Thank you.
 It's really nice of you to interview us and it's really great to meet you here.
 Thank you.