Author Interview - Memory-assisted prompt editing to improve GPT-3 after deployment
#nlp #gpt3 #prompt

This is an interview with the authors of this work, Aman Madaan and Niket Tandon.
Large language models such as GPT-3 have enabled many breakthroughs and new applications recently, but they come with an important downside: Training them is very expensive, and even fine-tuning is often difficult. This paper presents an adaptive method to improve performance of such models after deployment, without ever changing the model itself. This is done by maintaining a memory of interactions and then dynamically adapting new prompts by augmenting them with memory content. This has many applications, from non-intrusive fine-tuning to personalization.

OUTLINE:
0:00 - Intro
0:45 - Paper Overview
2:00 - What was your original motivation?
4:20 - There is an updated version of the paper!
9:00 - Have you studied this on real-world users?
12:10 - How does model size play into providing feedback?
14:10 - Can this be used for personalization?
16:30 - Discussing experimental results
17:45 - Can this be paired with recommender systems?
20:00 - What are obvious next steps to make the system more powerful?
23:15 - Clarifying the baseline methods
26:30 - Exploring cross-lingual customization
31:00 - Where did the idea for the clarification prompt come from?
33:05 - What did not work out during this project?
34:45 - What did you learn about interacting with large models?
37:30 - Final thoughts

Paper: https://arxiv.org/abs/2201.06009
Code & Data: https://github.com/madaan/memprompt

Abstract:
Large LMs such as GPT-3 are powerful, but can commit mistakes that are obvious to humans. For example, GPT-3 would mistakenly interpret "What word is similar to good?" to mean a homonym, while the user intended a synonym. Our goal is to effectively correct such errors via user interactions with the system but without retraining, which will be prohibitively costly. We pair GPT-3 with a growing memory of recorded cases where the model misunderstood the user's intents, along with user feedback for clarification. Such a memory allows our system to produce enhanced prompts for any new query based on the user feedback for error correction on similar cases in the past. On four tasks (two lexical tasks, two advanced ethical reasoning tasks), we show how a (simulated) user can interactively teach a deployed GPT-3, substantially increasing its accuracy over the queries with different kinds of misunderstandings by the GPT-3. Our approach is a step towards the low-cost utility enhancement for very large pre-trained LMs. All the code and data is available at this https URL.

Authors: Aman Madaan, Niket Tandon, Peter Clark, Yiming Yang

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, this is an interview with the authors of the paper on memory assisted prompt editing to improve GPT-3 after deployment. If you haven't seen it, I've made a comprehensive paper review on this paper and I released that yesterday. So the authors that I'm having on today as guests have seen that paper and we're able to dive right in. So if you haven't seen it, it might be a good place to check it out. I wish that you have a lot of fun following this interview or that you learn something or that you're entertained, ideally all three together. And yeah, have fun. Bye bye. Hi everyone. Today I'm here with Amon Madan and Niket Tandon of the paper Memory Assisted Prompt Editing to Improve GPT-3 After Deployment. Amon and Niket, thank you very much for being here. Welcome. Thank you for inviting me. So you've set out to write this paper and I guess the viewers have probably seen the review and this is really cool because these large language models, sure we now have a fine tuning endpoint at GPT-3. So it is a little bit possible to adjust it to your use case. But I think what you're doing right here comes the closest to what people imagine when they hear AI. Like when I go to someone and sell them an artificially like an AI system, they imagine a computer program that learns immediately, right? That they can tell things too and it adapts, it gets smarter as they interact with it. And largely the AI community has not delivered on that promise. We train things on static data sets and then we deploy them and they're frozen. And yet your system, I think, yeah, it comes the closest to really to live up to that promise. So I think that's really cool. How did you go? How did this come to be? How did you figure, you know, let's build something, let's build a plugin for GPT-3? Our original motivation was can we personalize very large models such as GPT-3 rather than having many copies of a giant GPT-3 model trained in one place on one static data along the way with the user, the models can improve, personalize over time. This was the original motivation why we started with this part. And GPT-3 was a great example to start with because it is such a large model that at the time of writing, it was not possible to fine tune these models. Yeah. So I think similar to that, one of the reasons why we specifically thought of having a plugin of software for GPT-3 is, so I was using copilot for some time and copilot makes the same mistake every time I write a print statement. So I'm using something like Python 3.7, which has f strings, which is a way of displaying the output, which you can nicely splice strings with variables. But the copilot will always use the older version of print statements. And I would have to go back, edit it and, you know, make it the f string that I want. So it was naturally, you know, kind of, there was this urge, you know, I wish there was something that could personalize this ID to me, but this instance of codecs to me. And you know, something like a hash map would work in that case. So whenever GPT-3 completes it with an older print statement, I can just have a regex that replaces the next string. And that kind of motivated this whole idea of having a small plugin outside of GPT-3 that stores these error cases and can correct them on the fly. And in the first version, we had some sort of proof of concept mixed up with kind of data. But the idea is to kind of not have to fail the model and having something super light that can exist to these things that not need to be repeated. Yeah, it's cool. And you don't even need to be open AI to do this, right? Because most research sort of assumes you're in control of the model. But this is really something you can just hang in front of whatever model that you're consuming, which is pretty cool. So I think, you know, it is important to say that I was quite critical of the paper in some places, and it's good to inform the viewers that there is actually a V2 out that addresses, I think, almost all of these criticisms in one batch. So I just quickly want to show that. And you told me that it got done like just in time last night or so. So there is a new version of the paper, which is on GitHub right now. I guess that's also coming on archive in the near future. And that does have a lot more experiments. Because I think one of the issues I had is that you said, well, we just want to present the framework of things. And you did some experiments. But can you maybe, you know, just talk about what new experiments you've added and how those turned out in this in this new version? Because if you know, with new experiments, and being state of the art, it is it sort of invalidates my point of, well, you just present only a framework. Yeah, so we did add like two different themes of tasks. One is ethical reasoning. And the other is more word reasoning. In ethical reasoning, this is a recent topic on ethical AI, which is as an example, if I have turned on the blender at 3am, I ask the system, is this ethically correct to do or not? And the system will probably should probably say that it is not okay to turn on your blender at 3am because it might disturb your neighbors. That's one theme, which is ethical, ethical AI. And we have two different tasks within that. In one case, the input is, you know, a string, like I said, turn on the blender at 3am, like a situation. And the output is whether it is good, bad or not. And like with some clarification, or some understanding, sorry, not clarification, just understanding of the model, why it believes this is the case. And we have two different types of understanding in it that makes up the two, you know, two different tasks. One is it clarifies it, the model presents its understanding based on an explanation of the sort that it's not good to wake up your neighbors or disturb your neighbors in the night. That's one. And the other setup we have, which makes up a different task is, you know, it says this is about care or harm. This is about, you know, the topic what this situation is intended to bring out. So that's one task, one theme of task. The other one is more word reasoning task. So we add on to the synthetic lexical relation task that we had in this, in the V1 paper. And we add on to word scrambling and other tasks, which are involving, you know, anagrams and how to fill up, how to correct a word misspelled and so on. So those are like two different themes of tasks we have. Aman, do you want to say something on the second task? I think we also added one other task, which is factual push answering. So suppose that user wants to ask factual questions like who is or where was a certain person born or where did they go to school? So things like that. So in those cases, there is no understanding that the model can display of the instruction other than the answer itself. So for example, if you ask where did Albert Einstein go to school, if the model says Stanford, then you can correct the model and say no, both ETS, UREC or something. And then you can store these corrections in the memory again. And then when you create the prompt, you would bring in some examples which are similar to the question on this the model has been wrong before to make the prompt. So for example, if the question comes in where did Winston Churchill go to school, then you would already have an example of the Albert Einstein example. And that we show is helping the model getting better at these tasks. So two different themes, the two layer and factual questions. Have you so? Yeah, so this is pretty cool. And I've had a flick through this paper that it the tasks seem to be much more extensive. Now, that's not it. It's a so you had the ethical one, you give a few examples right here. On the right, we can see, for example, the understanding this question is about loving your partner, this question about seeking medical attention, if you feel there's something wrong, which is a lot, I think, you know, the the gap to what we what people usually call common sense gets smaller and smaller. Have you let any users any actual users use this system with GPT three, so you came up with your own data set as if I understand correctly, your own sort of feedback, sometimes heuristics and so on. Did you ever just, you know, set this in front of someone and say, you know, here you go, try it out? No, we have not. That's one of the things we would like to do. So we have not done that yet. And in fact, in just to clarify, the the data sets that we have here are the feedbacks on ethical reasoning, for example, is not something that we came up with. This was present in the data itself. So this was a data which was crowdsourced through mechanical torque. And there were actual users who are actual mechanical turkers who gave this feedback. But on the other hand, we have not tried this on any real users. This is the closest we came to reality in some sense. But we would like to do this in the future. Yeah, it'd be super cool to see how real people interact with this. Sorry, Aman. Yeah, so I think so like Nikit said that for both these data sets, the data set is real. So you're right in the first version, we had one of the data sets that we collected ourselves. But in this case, the feedback is given by humans. So in some sense, we are approximating that process by a linear data collection process as opposed to a bunch of workers working on it at the same time. But yes, it would be great to kind of see if you know, once deployed, if this actually does better on one of these tasks or one of the new tasks that we discussed. I'm going to guess that specifically for GPT-3, the restriction of OpenAI on what you can build with it and the approval process would prevent you from actually releasing this, say to the public as a service. But one could think of maybe using another model or just I mean, your code is online. So people could use it with their own API key if they really wanted to. Yeah, that is correct. And in fact, just outside of this paper also, we had been working on T5 model with a very similar architecture, T511B. And so that's one of the models we could release in the future. Is there a difference between smaller models and larger models in how much this type of feedback is needed? Like you specifically work with GPT-3 and you know, I get it, that's the model that we cannot train. But is it also more necessary to provide feedback? Can you tell us a little bit about the differences between small and large models or different models? Let me just start with that. So it's a really good question, first of all. So our general experience with injecting, you know, some knowledge, external knowledge, like you know, common sense knowledge into models has been as the model capacity keeps increasing, it requires comparatively less knowledge injection. So smaller models like, you know, let's say Bard-Base would require, would benefit a lot by we have seen this in the experiments in the past on, and others have also reported it. If you inject external common sense knowledge, then those models get much bigger boost than for example, T511B. Bigger models get less boost. So we have tried the same, very similar architecture, actually almost the same architecture, there's a paper under review on T511B. And what we also observed there is that there is substantial gains with T511B. The only difference in mechanism is that, you know, there we were able to fine tune, have a fine tune T5 model, which understands the task a lot better than in GPT-3 where there was not even an opportunity to do that. So probably because of that reason, we are seeing bigger boost in GPT-3 than we did with T511B. But in both the cases, there is substantial boost in performance by doing so. Cool. And have you tried, so what you are doing right here, it goes very much into the direction of correcting the model if it, let's say, makes a mistake, or if it misunderstands something. I had the sort of the opinion that personalization, very much in the sense of how you, Amon, said this before, you know, I want my IDE to do something in a particular way, would benefit hugely from that. Is this something on your mind too? Are you looking into various like personalization aspects of these models? Or is this something that is for some reason not possible? Yeah, I think that's a very good point. And in fact, in the first version, in this version, we have some experiments in the amendments, also in the earlier version, where we simulate users who sort of interact with the model in Hindi or Punjabi. And that's some sort of personalization, it's kind of a language personalization. So there's a person who's speaking in a dialect of Hindi or Punjabi, and even there's a certain phrase they use to be pp. And if you can store that in memory, then sure, the first time the model is not mitigated, but the next time someone comes and uses the same word, you know, hopefully it will be patched. So we did kind of create some experiments on that angle. And we also have examples in the ethical AI setting where the model was able to correct or kind of work with slang usage. When people were saying the same thing in slangs, right, so one person comes and they give feedback. So I think it's a very promising direction for personalization. And I anticipate that in the near future, systems that are doing successfully to do this in their architecture, but they have this memory that kind of has an impact. If we get into the paper a little bit, like into a bit more sort of the technical aspects here, I want to jump over to the experiment section. And you had an interesting plot where you show not this one, not this one. This one is one of them. An interesting, no, this is the outer vocabulary. I think the main ones are I missed them. Oh, here, I've drawn so much over them that it's, it's a mess. Specifically, I was I was wondering this PFB of 0.5. Did I interpret this correctly, that this means that you only get the feedback half of the time? Does that mean the user can only give feedback half of the time? Or the model only receives sort of this feedback or the model only gets to go through this feedback loop half of the time? The user gives feedback. Okay, because then the memory grows slowly. Then it makes total sense that they end up sort of converging to the same place because I was wondering, you know, if if your procedure was only active half the time, it should fail half the time. But if the user is able to give feedback half the time, it would still learn slowly, but it would still learn over time. Okay, that's we wanted to simulate reluctant users who might, you know, not always give feedback. So yeah, sometimes you want to give feedback, sometimes not. Yeah. Have you have you thought about pairing this with recommender systems? Because in recommender system, sort of a recommender system would group me together with other users who have like similar preferences as I do. So you know, conceivably, I could say, well, maybe I'm able to sort of profit off of feedback of those users, right? If I if I give some feedback, and I'm very similar to these users, it might be the same. Is this something that that could be done? Or? Yeah, I think this is a really neat idea. We did not think about it, but now that I think about it, when you mentioned it, I think it is a it makes total sense to have a community of similar users, all having, you know, similar preferences. It makes total sense. And I think it would be very cool to try this in the future. Well maybe or you always know who the feedback comes from is like, ah, your dumb friend entered. It's yeah, I think I'm thinking of these people who enter, who like, altogether enter dumb things into Google so that Google auto complete suggests the dumb thing. You know, that brings to a very good point about sabotaging our system. It is possible. I mean, if you keep giving it really bad feedback, eventually it is going to apply bad feedback to, you know, newer examples. And this is a valid point, a valid concern. We also don't know if our memory can be consistent over time or can start deteriorating and becoming like inconsistent among itself. You know, I could just give different examples with different feedbacks. So there is not not our work, but there has been other work on, you know, how to maintain consistency in a memory over time. But that's an additional direction of research which we can employ within our system to keep it healthy and consistent. Are there you another in another point in the paper, you mentioned these different pieces of the puzzle in this framework you you propose. You've added more tasks. Have you also thought about amending or augmenting some of these things to be more, let's say more complicated, maybe replace some stuff with learn things so far you have to look up which is a language model or an embedding model. Yet the other pieces of the puzzle here are fairly simple so far in your experiments. Are there any obvious next steps where to make this more powerful in any of these four parts? Yeah, so that is true. In fact, the current implementation is for the combiner is as simple as you know, it's just a threshold is just thresholding over the inner product. You know, it's that simple. But eventually we are in the process. So this is very much work in progress where we are trying to, you know, beef up the other components also. Right now our only focus was on look up and memory and the other components are very simple. But eventually this is where we are getting at, you know, work in progress. And I think there are lots of lots of details where you know, our current system is very primitive in the sense that it it only assumes that the users are, you know, really nice and that they don't give you bad feedback. That's one. It also assumes that the users can, you know, you can effectively retrieve from the past. And that's not always the case. You know, we there are cases where we are not able to do that. That's why we had to set, you know, a higher threshold where we we only get good good matches and like good feedback, which are very similar. But you know, something which we would like to do and look up, I'm just giving an example. It's like suppose your input is turn on the blender at 3am and now a new input comes in, which is saying playing drums late night. You know, both of them are in the analogy space of errors. They're actually very similar, but that's not something which our current system can match. It can at most say, oh, well, if if I find something like turn on the mixer at 2am, that's similar to something I found and it will pick that feedback, you know. So this kind of really recursive reminding to a model based on similar error space is the next step where we are getting to with this lookup. I think also in the space of the combiner and the prompter specifically, there is probably a lot of potential to still be gained. I mean, instead of concatenating, you could you could imagine any, you know, many smart ways of combining what you retrieve from the memory with what you already have. Potentially, you could even ask the model itself to come up with sort of like a better prompt or to sort of you can maybe abuse the model again to suggest better things to you. I mean, I think that the possibilities are are quite quite open here to make this very, very cool, very powerful. Another thing that I wasn't sure about is your baseline, this grow prompt baseline right here. And I think I tried to explain this a little bit. Do I understand correctly that the grow prompt baseline, you take whatever the contents of your memory are and you just append them to the prompt before the question? Okay. Yeah, my concern was a little bit that it's not exactly right that the baseline because the prompt is structured differently. But I don't know how important that ultimately will be. Probably not. So I think we do structure the prompt in the same fashion. So we get examples and the structure of the prompt does not change. It's just like a longer prompt. So in the video you show an example prompt which is in the appendix. It's the same format. It's just much longer. It's basically as much as we can fit. So wait, we can look at one here. So this is the entire prompt, which I found pretty cool that not only do you prime the model to sort of give you the answers and give you the understanding, which is, you know, that's I think that's pretty cool idea in itself to get side information with your main information out of these models that you can then use to query them again. I think the applications for this are much larger than just this one. You also train the model to specifically view or regard or pay attention to the clarifications. My question was that, let's, this is a bit fat. When in your main method, when you retrieve a clarification, do I see this correctly that you append it at the end right here to the question currently? And this grow sort of this baseline would append something like here in between? Or do I see this incorrectly? Right. So in the grow prompt, what we do is we essentially add more examples to the prompt. So instead of retrieving something from the maybe it's added to the prompt itself. Yeah. Okay. So that's cool. Yeah. Then I've understood correctly. Sorry. The mechanism is kind of very similar to our own methods, sort of like, you know, retrieve the right feedback in some sense. The only thing is we now we are allowing GPT-3 to attend over those, to attend over it rather than, you know, be providing a retrieval function from the memory. We hope that GPT-3 will be able to attend over it itself. Yes. I mean, yeah. And if it fits into the prompt, it's pretty certain that at least it might pick up on it. Right. And you make good points here. You say that this grow prompt, it is quite a bit larger and it cannot scale up. So as soon as things fall out of your memory without a good retrieval function, you're essentially limited to a very short time horizon. There is this experiment here, this plot right here, which I haven't touched at all, which it goes a little bit into out of vocabulary domain, a little bit into the domain of different languages, maybe lower resource languages. Do you want to comment a little bit on what you, what you did there and what your findings were? Yeah, so the idea is essentially very similar to what I was talking about earlier. So the prompt itself has examples from Hindi, for example, and then the questions also come in Hindi. And, you know, for the first time around when the question comes, GPT-3 would not know because it's primarily English. Funny thing is for Hindi actually, sometimes it gets it. Or apparently there's lots of, you know, English, English corpus online. But for Punjabi it struggles. So the idea is the user comes in and does something, the model doesn't get it, it goes in the memory, next time something comes as a similar question. So the model retrieves the understanding from the memory and hopefully is able to do the test. So to clarify that the questions are in Punjabi, for example, that you would like to have answered. And you also construct a prompt in Punjabi or is the prompt still in English? The prompt is transcribed in English, but the quotient parts are all in Punjabi. So the script is not the Punjabi script. It's still English, but parts of it are in Punjabi. So we have an example in the appendix. Yeah. Oh, yeah, that's a good point. We should go. It's, yeah. No? Yeah, so I think one of those. This is the end right here. I think this one might be. Yeah, so those are in Hindi and the one in the bottom is in Punjabi. So the person is, you know, trying to, the scenario I had in 907 is trying to learn English and they're trying to look up words. So in the first case, they are saying, what is the opposite of edit? So they say, they ask it in Punjabi. So they know that they want meaning of this word edit and the rest of it, they ask in Punjabi and the model says something that the opposite of this is something else. And then the person can say, no, I want synonyms. And there's like one missing piece here, which is that you have to tell the user, and then means opposite in Punjabi. So they know what the model is, you know, it's trying to say. Okay, so you could interact with this thing sort of across languages and you could prime it to say, which parts do I want in which language? Because it would obviously not know, I guess, what you want the answer in. Yeah, yeah, you can definitely add language tags and that could definitely be it. I mean, it's a pretty cool example of exactly of personalization, right? Because you can imagine you personalize this exactly to sort of how you want to interact with it. And someone else who might be more or less skilled at English or in reverse in Punjabi might do a different thing. That's pretty cool. Yeah, there's one more point I wanted to mention which you kind of mentioned earlier with respect to the prompt. So as you noticed in our prompt, the model does not only give out the answer, it also gives out its understanding of the question. And I think that's a very kind of crucial piece in this design, because one of the bottlenecks for us earlier was the system, a system that is used that the user knows the real answer is not really practical because if the user knew the answer, they would be playing with the model right outside of an annotation setting. So this kind of breaks that barrier. So you might not know what the answer is, but you know for sure what you ask for. So you can always tell the model, no, this is not what I don't know if you're right, but I know for sure this is not what I want. And that kind of helps in improving the performance. The performance of the model itself might be whatever it is, but we are helping the model in understanding that intent more precisely. That's the main trick here. Yeah I like this getting the answer with the understanding. I think that's pretty powerful, not only to interact with the model, but also just to understand what it does instead of just getting a simple answer. It could be a good recipe for other applications as well. Did you have to fiddle around a lot with sort of the prompt structure or the structure of what to add? Right now you have a bar and then clarification and then colon. Is this the first try and it worked or is this the result of many hours of sweat and tears? No, so it's a first try and we did not impose any intention because our goal was not to show our game. The goal was to give it words. And you know this weird hash and new line. This is what we took from OpenAS website. They had a bunch of instructions on best practices for formatting your prompt. I think they have changed it since, but we just took it from OpenAS website. And this was also one of the main motivations like even if I don't know how to exactly have the prompts here, there are two ways in which you could gain improvements here. One is in context examples within the prompt and the other is at the question side. There are like just two aspects for fiddling with this. And there has been a lot of work on how to give the right in context examples, what order, what examples, how to select them. Our focus is on the question part, like only on the input part which comes from the user. And we are trying to pull all the knobs, like turn all the knobs at that end and in some sense we were able to overcome some limitations which our prompts probably have. Maybe there are much better ways of coming up with a prompt than we have. But I think all those methods are just, if we plug in any of the nicer methods to come up with a better prompt, that's just icing on the cake for us. If this was first try and it's still in there, so obviously it worked, was there things that didn't work out over the course of this research? Like things where you got stuck or maybe even ideas that you had to discard halfway through? I can tell one which really bothered us for a long time. It's on contrastive prompting, which is we wanted to also give negative answers. Can the user just say, no, that's not the right answer. With autoregressive models, it is really difficult to somehow give them steer away from probability mass towards certain tokens. It's really difficult to do that. We are still not able to effectively do that. Ideally, in the real world, users will give, I think users will give feedback of the kind instead of clarifications. In addition to clarification, they can also say, no, this is not right or this is why it's not right. The model came up with what's the capital of India and it says the capital is Mumbai. I just want to say, no, it is not. It is like Delhi or like you're looking at the wrong places. That's something which we were not able to do. I think it's an open problem, like this kind of negative prompting. It's valuable from a feedback perspective for the future. We just don't know how to solve it right now. What did you do? You played obviously a little bit with these large models with the API, presumably also tried out yourself a lot of things I can only assume over the course of this research. Is there anything maybe also a bit independent of the research itself? Is there anything that you came across that surprised you about these large models and how people can interact with them? I think for me, one of the things that XB stood out from early days is how good copilot was. I think if you really have been using it on a day to day basis, and I have been using it for a few months now, it has consistently gotten better. Initially it had these small weird quirks. These models basically generate left to right or top to bottom. If I have some, but when you program, you would write some functions below and then you go back up to a function and you want to reference the function below. So that did not work earlier. So it would only condition on things that it had seen so far in the file. But they have improved the whole that stuff also. So I think it's astonishing that at least in the structure setting, how good they are for generating things. At the same time, it's also interesting that even when you have 175 billion parameters, how poor the model is at common sense, because it's very clear when you go from these structured settings to a more open ended setting, the common sense generation or common sense medium, I still think the models struggle a lot. So it still is clear that there's a long way to go. So there's a bit of hope. So I think you have to choose your end application wisely. But there are clearly very cool applications that can be built for which you don't need AGI, as long as you have a very good pattern manager. One of the surprises for me was on like just the fact that these models are correctable, you know, like a model can make mistakes which are hopeless, you know, it's just total understanding is wrong. But I think over time, what has happened is with larger models, even though there might be many claims that it is missing common sense, and it is, you know, these models are dumb and so on. But I do believe that, you know, for a certain question, yes, there might be cases where it's not coming up with the right answer, but they're still correctable. They're not dumb anymore. I think these models are getting they're correctable in the sense that their output is not completely off and with some guidance, they can get to the right answer. Awesome. Is there something other than that, that you feel I have maybe not touched in my review that you would like viewers to know or, you know, be able to understand or anything that I've maybe gotten wrong? I think most of the stuff you said was correct. Like it was nothing was wrong, really. Your understanding and almost everything was was correct. Just the only thing I'm not I'm not fishing for compliments. Legitimately, if there's something that you feel like, you know, people should know about this that we haven't talked about at all. Yeah, yeah. I think the part about that you mentioned in your video about the feedback could be misleading. I think we'd be best upon it. But I think that's a valid criticism that still holds. And that was one of the things that we have not been able to solve even now. So we are we are we are trying different kinds of retrieval conditioning on the expected output doing something like you said, more complex in one of those four modules. But I think that remains a valid criticism of the work that there would be cases where feedback would distract. So the model was going to say the right thing, but because you have this thing, it's saying the wrong thing. But we think that problem is kind of there's an easier to solve it is it's to show both the answers to the user and let the user pick one. So we show this is the answer that I would have given you. This is what I would give you with some feedback. Pick one. But if you don't want to do that, then it's kind of very challenging because the model somehow has to know that it's going to make a mistake and only then it's it should pull up feedback, etc. And those are kind of having it's very hard for models to know that they're wrong or to know what they don't know. So that's a big challenge and kind of one interesting research direction that we are pursuing outside of this, which is how can we let a model know that they don't know or then start it going wrong and what can we do in those cases? I agree. And if you can, if you can do that with a model that you don't even have access to, I think that would be a little bit of a little bit of a grail of research. That would be seriously cool. And I think it would it would improve a lot of applications of these models around, you know, all around technology. Cool. Well, Niket and Aman, thank you very much for being here. It was a pleasure. And I hope this work goes on and becomes more powerful over time. Thanks, Henrik. Thank you. Thank you so much for having us. Thank you.
 Hello, this is an interview with the authors of the paper on memory assisted prompt editing
 to improve GPT-3 after deployment.
 If you haven't seen it, I've made a comprehensive paper review on this paper and I released
 that yesterday.
 So the authors that I'm having on today as guests have seen that paper and we're able
 to dive right in.
 So if you haven't seen it, it might be a good place to check it out.
 I wish that you have a lot of fun following this interview or that you learn something
 or that you're entertained, ideally all three together.
 And yeah, have fun.
 Bye bye.
 Hi everyone.
 Today I'm here with Amon Madan and Niket Tandon of the paper Memory Assisted Prompt
 Editing to Improve GPT-3 After Deployment.
 Amon and Niket, thank you very much for being here.
 Welcome.
 Thank you for inviting me.
 So you've set out to write this paper and I guess the viewers have probably seen the
 review and this is really cool because these large language models, sure we now have a
 fine tuning endpoint at GPT-3.
 So it is a little bit possible to adjust it to your use case.
 But I think what you're doing right here comes the closest to what people imagine when they
 hear AI.
 Like when I go to someone and sell them an artificially like an AI system, they imagine
 a computer program that learns immediately, right?
 That they can tell things too and it adapts, it gets smarter as they interact with it.
 And largely the AI community has not delivered on that promise.
 We train things on static data sets and then we deploy them and they're frozen.
 And yet your system, I think, yeah, it comes the closest to really to live up to that promise.
 So I think that's really cool.
 How did you go?
 How did this come to be?
 How did you figure, you know, let's build something, let's build a plugin for GPT-3?
 Our original motivation was can we personalize very large models such as GPT-3 rather than
 having many copies of a giant GPT-3 model trained in one place on one static data along
 the way with the user, the models can improve, personalize over time.
 This was the original motivation why we started with this part.
 And GPT-3 was a great example to start with because it is such a large model that at the
 time of writing, it was not possible to fine tune these models.
 Yeah.
 So I think similar to that, one of the reasons why we specifically thought of having a plugin
 of software for GPT-3 is, so I was using copilot for some time and copilot makes the same mistake
 every time I write a print statement.
 So I'm using something like Python 3.7, which has f strings, which is a way of displaying
 the output, which you can nicely splice strings with variables.
 But the copilot will always use the older version of print statements.
 And I would have to go back, edit it and, you know, make it the f string that I want.
 So it was naturally, you know, kind of, there was this urge, you know, I wish there was
 something that could personalize this ID to me, but this instance of codecs to me.
 And you know, something like a hash map would work in that case.
 So whenever GPT-3 completes it with an older print statement, I can just have a regex that
 replaces the next string.
 And that kind of motivated this whole idea of having a small plugin outside of GPT-3
 that stores these error cases and can correct them on the fly.
 And in the first version, we had some sort of proof of concept mixed up with kind of
 data.
 But the idea is to kind of not have to fail the model and having something super light
 that can exist to these things that not need to be repeated.
 Yeah, it's cool.
 And you don't even need to be open AI to do this, right?
 Because most research sort of assumes you're in control of the model.
 But this is really something you can just hang in front of whatever model that you're
 consuming, which is pretty cool.
 So I think, you know, it is important to say that I was quite critical of the paper in
 some places, and it's good to inform the viewers that there is actually a V2 out that addresses,
 I think, almost all of these criticisms in one batch.
 So I just quickly want to show that.
 And you told me that it got done like just in time last night or so.
 So there is a new version of the paper, which is on GitHub right now.
 I guess that's also coming on archive in the near future.
 And that does have a lot more experiments.
 Because I think one of the issues I had is that you said, well, we just want to present
 the framework of things.
 And you did some experiments.
 But can you maybe, you know, just talk about what new experiments you've added and how
 those turned out in this in this new version?
 Because if you know, with new experiments, and being state of the art, it is it sort
 of invalidates my point of, well, you just present only a framework.
 Yeah, so we did add like two different themes of tasks.
 One is ethical reasoning.
 And the other is more word reasoning.
 In ethical reasoning, this is a recent topic on ethical AI, which is as an example, if
 I have turned on the blender at 3am, I ask the system, is this ethically correct to do
 or not?
 And the system will probably should probably say that it is not okay to turn on your blender
 at 3am because it might disturb your neighbors.
 That's one theme, which is ethical, ethical AI.
 And we have two different tasks within that.
 In one case, the input is, you know, a string, like I said, turn on the blender at 3am, like
 a situation.
 And the output is whether it is good, bad or not.
 And like with some clarification, or some understanding, sorry, not clarification, just
 understanding of the model, why it believes this is the case.
 And we have two different types of understanding in it that makes up the two, you know, two
 different tasks.
 One is it clarifies it, the model presents its understanding based on an explanation
 of the sort that it's not good to wake up your neighbors or disturb your neighbors in
 the night.
 That's one.
 And the other setup we have, which makes up a different task is, you know, it says this
 is about care or harm.
 This is about, you know, the topic what this situation is intended to bring out.
 So that's one task, one theme of task.
 The other one is more word reasoning task.
 So we add on to the synthetic lexical relation task that we had in this, in the V1 paper.
 And we add on to word scrambling and other tasks, which are involving, you know, anagrams
 and how to fill up, how to correct a word misspelled and so on.
 So those are like two different themes of tasks we have.
 Aman, do you want to say something on the second task?
 I think we also added one other task, which is factual push answering.
 So suppose that user wants to ask factual questions like who is or where was a certain
 person born or where did they go to school?
 So things like that.
 So in those cases, there is no understanding that the model can display of the instruction
 other than the answer itself.
 So for example, if you ask where did Albert Einstein go to school, if the model says
 Stanford, then you can correct the model and say no, both ETS, UREC or something.
 And then you can store these corrections in the memory again.
 And then when you create the prompt, you would bring in some examples which are similar to
 the question on this the model has been wrong before to make the prompt.
 So for example, if the question comes in where did Winston Churchill go to school, then you
 would already have an example of the Albert Einstein example.
 And that we show is helping the model getting better at these tasks.
 So two different themes, the two layer and factual questions.
 Have you so?
 Yeah, so this is pretty cool.
 And I've had a flick through this paper that it the tasks seem to be much more extensive.
 Now, that's not it.
 It's a so you had the ethical one, you give a few examples right here.
 On the right, we can see, for example, the understanding this question is about loving
 your partner, this question about seeking medical attention, if you feel there's something
 wrong, which is a lot, I think, you know, the the gap to what we what people usually
 call common sense gets smaller and smaller.
 Have you let any users any actual users use this system with GPT three, so you came up
 with your own data set as if I understand correctly, your own sort of feedback, sometimes
 heuristics and so on.
 Did you ever just, you know, set this in front of someone and say, you know, here you go,
 try it out?
 No, we have not.
 That's one of the things we would like to do.
 So we have not done that yet.
 And in fact, in just to clarify, the the data sets that we have here are the feedbacks on
 ethical reasoning, for example, is not something that we came up with.
 This was present in the data itself.
 So this was a data which was crowdsourced through mechanical torque.
 And there were actual users who are actual mechanical turkers who gave this feedback.
 But on the other hand, we have not tried this on any real users.
 This is the closest we came to reality in some sense.
 But we would like to do this in the future.
 Yeah, it'd be super cool to see how real people interact with this.
 Sorry, Aman.
 Yeah, so I think so like Nikit said that for both these data sets, the data set is real.
 So you're right in the first version, we had one of the data sets that we collected ourselves.
 But in this case, the feedback is given by humans.
 So in some sense, we are approximating that process by a linear data collection process
 as opposed to a bunch of workers working on it at the same time.
 But yes, it would be great to kind of see if you know, once deployed, if this actually
 does better on one of these tasks or one of the new tasks that we discussed.
 I'm going to guess that specifically for GPT-3, the restriction of OpenAI on what you can
 build with it and the approval process would prevent you from actually releasing this,
 say to the public as a service.
 But one could think of maybe using another model or just I mean, your code is online.
 So people could use it with their own API key if they really wanted to.
 Yeah, that is correct.
 And in fact, just outside of this paper also, we had been working on T5 model with a very
 similar architecture, T511B.
 And so that's one of the models we could release in the future.
 Is there a difference between smaller models and larger models in how much this type of
 feedback is needed?
 Like you specifically work with GPT-3 and you know, I get it, that's the model that
 we cannot train.
 But is it also more necessary to provide feedback?
 Can you tell us a little bit about the differences between small and large models or different
 models?
 Let me just start with that.
 So it's a really good question, first of all.
 So our general experience with injecting, you know, some knowledge, external knowledge,
 like you know, common sense knowledge into models has been as the model capacity keeps
 increasing, it requires comparatively less knowledge injection.
 So smaller models like, you know, let's say Bard-Base would require, would benefit a lot
 by we have seen this in the experiments in the past on, and others have also reported
 it.
 If you inject external common sense knowledge, then those models get much bigger boost than
 for example, T511B.
 Bigger models get less boost.
 So we have tried the same, very similar architecture, actually almost the same architecture, there's
 a paper under review on T511B.
 And what we also observed there is that there is substantial gains with T511B.
 The only difference in mechanism is that, you know, there we were able to fine tune,
 have a fine tune T5 model, which understands the task a lot better than in GPT-3 where
 there was not even an opportunity to do that.
 So probably because of that reason, we are seeing bigger boost in GPT-3 than we did with
 T511B.
 But in both the cases, there is substantial boost in performance by doing so.
 Cool.
 And have you tried, so what you are doing right here, it goes very much into the direction
 of correcting the model if it, let's say, makes a mistake, or if it misunderstands something.
 I had the sort of the opinion that personalization, very much in the sense of how you, Amon, said
 this before, you know, I want my IDE to do something in a particular way, would benefit
 hugely from that.
 Is this something on your mind too?
 Are you looking into various like personalization aspects of these models?
 Or is this something that is for some reason not possible?
 Yeah, I think that's a very good point.
 And in fact, in the first version, in this version, we have some experiments in the amendments,
 also in the earlier version, where we simulate users who sort of interact with the model
 in Hindi or Punjabi.
 And that's some sort of personalization, it's kind of a language personalization.
 So there's a person who's speaking in a dialect of Hindi or Punjabi, and even there's a certain
 phrase they use to be pp.
 And if you can store that in memory, then sure, the first time the model is not mitigated,
 but the next time someone comes and uses the same word, you know, hopefully it will be
 patched.
 So we did kind of create some experiments on that angle.
 And we also have examples in the ethical AI setting where the model was able to correct
 or kind of work with slang usage.
 When people were saying the same thing in slangs, right, so one person comes and they
 give feedback.
 So I think it's a very promising direction for personalization.
 And I anticipate that in the near future, systems that are doing successfully to do
 this in their architecture, but they have this memory that kind of has an impact.
 If we get into the paper a little bit, like into a bit more sort of the technical aspects
 here, I want to jump over to the experiment section.
 And you had an interesting plot where you show not this one, not this one.
 This one is one of them.
 An interesting, no, this is the outer vocabulary.
 I think the main ones are I missed them.
 Oh, here, I've drawn so much over them that it's, it's a mess.
 Specifically, I was I was wondering this PFB of 0.5.
 Did I interpret this correctly, that this means that you only get the feedback half
 of the time?
 Does that mean the user can only give feedback half of the time?
 Or the model only receives sort of this feedback or the model only gets to go through this
 feedback loop half of the time?
 The user gives feedback.
 Okay, because then the memory grows slowly.
 Then it makes total sense that they end up sort of converging to the same place because
 I was wondering, you know, if if your procedure was only active half the time, it should fail
 half the time.
 But if the user is able to give feedback half the time, it would still learn slowly, but
 it would still learn over time.
 Okay, that's we wanted to simulate reluctant users who might, you know, not always give
 feedback.
 So yeah, sometimes you want to give feedback, sometimes not.
 Yeah.
 Have you have you thought about pairing this with recommender systems?
 Because in recommender system, sort of a recommender system would group me together with other
 users who have like similar preferences as I do.
 So you know, conceivably, I could say, well, maybe I'm able to sort of profit off of feedback
 of those users, right?
 If I if I give some feedback, and I'm very similar to these users, it might be the same.
 Is this something that that could be done?
 Or?
 Yeah, I think this is a really neat idea.
 We did not think about it, but now that I think about it, when you mentioned it, I think
 it is a it makes total sense to have a community of similar users, all having, you know, similar
 preferences.
 It makes total sense.
 And I think it would be very cool to try this in the future.
 Well maybe or you always know who the feedback comes from is like, ah, your dumb friend entered.
 It's yeah, I think I'm thinking of these people who enter, who like, altogether enter dumb
 things into Google so that Google auto complete suggests the dumb thing.
 You know, that brings to a very good point about sabotaging our system.
 It is possible.
 I mean, if you keep giving it really bad feedback, eventually it is going to apply bad feedback
 to, you know, newer examples.
 And this is a valid point, a valid concern.
 We also don't know if our memory can be consistent over time or can start deteriorating and becoming
 like inconsistent among itself.
 You know, I could just give different examples with different feedbacks.
 So there is not not our work, but there has been other work on, you know, how to maintain
 consistency in a memory over time.
 But that's an additional direction of research which we can employ within our system to keep
 it healthy and consistent.
 Are there you another in another point in the paper, you mentioned these different pieces
 of the puzzle in this framework you you propose.
 You've added more tasks.
 Have you also thought about amending or augmenting some of these things to be more, let's say
 more complicated, maybe replace some stuff with learn things so far you have to look
 up which is a language model or an embedding model.
 Yet the other pieces of the puzzle here are fairly simple so far in your experiments.
 Are there any obvious next steps where to make this more powerful in any of these four
 parts?
 Yeah, so that is true.
 In fact, the current implementation is for the combiner is as simple as you know, it's
 just a threshold is just thresholding over the inner product.
 You know, it's that simple.
 But eventually we are in the process.
 So this is very much work in progress where we are trying to, you know, beef up the other
 components also.
 Right now our only focus was on look up and memory and the other components are very simple.
 But eventually this is where we are getting at, you know, work in progress.
 And I think there are lots of lots of details where you know, our current system is very
 primitive in the sense that it it only assumes that the users are, you know, really nice
 and that they don't give you bad feedback.
 That's one.
 It also assumes that the users can, you know, you can effectively retrieve from the past.
 And that's not always the case.
 You know, we there are cases where we are not able to do that.
 That's why we had to set, you know, a higher threshold where we we only get good good matches
 and like good feedback, which are very similar.
 But you know, something which we would like to do and look up, I'm just giving an example.
 It's like suppose your input is turn on the blender at 3am and now a new input comes in,
 which is saying playing drums late night.
 You know, both of them are in the analogy space of errors.
 They're actually very similar, but that's not something which our current system can
 match.
 It can at most say, oh, well, if if I find something like turn on the mixer at 2am, that's
 similar to something I found and it will pick that feedback, you know.
 So this kind of really recursive reminding to a model based on similar error space is
 the next step where we are getting to with this lookup.
 I think also in the space of the combiner and the prompter specifically, there is probably
 a lot of potential to still be gained.
 I mean, instead of concatenating, you could you could imagine any, you know, many smart
 ways of combining what you retrieve from the memory with what you already have.
 Potentially, you could even ask the model itself to come up with sort of like a better
 prompt or to sort of you can maybe abuse the model again to suggest better things to you.
 I mean, I think that the possibilities are are quite quite open here to make this very,
 very cool, very powerful.
 Another thing that I wasn't sure about is your baseline, this grow prompt baseline right
 here.
 And I think I tried to explain this a little bit.
 Do I understand correctly that the grow prompt baseline, you take whatever the contents of
 your memory are and you just append them to the prompt before the question?
 Okay.
 Yeah, my concern was a little bit that it's not exactly right that the baseline because
 the prompt is structured differently.
 But I don't know how important that ultimately will be.
 Probably not.
 So I think we do structure the prompt in the same fashion.
 So we get examples and the structure of the prompt does not change.
 It's just like a longer prompt.
 So in the video you show an example prompt which is in the appendix.
 It's the same format.
 It's just much longer.
 It's basically as much as we can fit.
 So wait, we can look at one here.
 So this is the entire prompt, which I found pretty cool that not only do you prime the
 model to sort of give you the answers and give you the understanding, which is, you
 know, that's I think that's pretty cool idea in itself to get side information with your
 main information out of these models that you can then use to query them again.
 I think the applications for this are much larger than just this one.
 You also train the model to specifically view or regard or pay attention to the clarifications.
 My question was that, let's, this is a bit fat.
 When in your main method, when you retrieve a clarification, do I see this correctly that
 you append it at the end right here to the question currently?
 And this grow sort of this baseline would append something like here in between?
 Or do I see this incorrectly?
 Right.
 So in the grow prompt, what we do is we essentially add more examples to the prompt.
 So instead of retrieving something from the maybe it's added to the prompt itself.
 Yeah.
 Okay.
 So that's cool.
 Yeah.
 Then I've understood correctly.
 Sorry.
 The mechanism is kind of very similar to our own methods, sort of like, you know, retrieve
 the right feedback in some sense.
 The only thing is we now we are allowing GPT-3 to attend over those, to attend over it rather
 than, you know, be providing a retrieval function from the memory.
 We hope that GPT-3 will be able to attend over it itself.
 Yes.
 I mean, yeah.
 And if it fits into the prompt, it's pretty certain that at least it might pick up on
 it.
 Right.
 And you make good points here.
 You say that this grow prompt, it is quite a bit larger and it cannot scale up.
 So as soon as things fall out of your memory without a good retrieval function, you're
 essentially limited to a very short time horizon.
 There is this experiment here, this plot right here, which I haven't touched at all, which
 it goes a little bit into out of vocabulary domain, a little bit into the domain of different
 languages, maybe lower resource languages.
 Do you want to comment a little bit on what you, what you did there and what your findings
 were?
 Yeah, so the idea is essentially very similar to what I was talking about earlier.
 So the prompt itself has examples from Hindi, for example, and then the questions also come
 in Hindi.
 And, you know, for the first time around when the question comes, GPT-3 would not know because
 it's primarily English.
 Funny thing is for Hindi actually, sometimes it gets it.
 Or apparently there's lots of, you know, English, English corpus online.
 But for Punjabi it struggles.
 So the idea is the user comes in and does something, the model doesn't get it, it goes
 in the memory, next time something comes as a similar question.
 So the model retrieves the understanding from the memory and hopefully is able to do the
 test.
 So to clarify that the questions are in Punjabi, for example, that you would like to have answered.
 And you also construct a prompt in Punjabi or is the prompt still in English?
 The prompt is transcribed in English, but the quotient parts are all in Punjabi.
 So the script is not the Punjabi script.
 It's still English, but parts of it are in Punjabi.
 So we have an example in the appendix.
 Yeah.
 Oh, yeah, that's a good point.
 We should go.
 It's, yeah.
 No?
 Yeah, so I think one of those.
 This is the end right here.
 I think this one might be.
 Yeah, so those are in Hindi and the one in the bottom is in Punjabi.
 So the person is, you know, trying to, the scenario I had in 907 is trying to learn English
 and they're trying to look up words.
 So in the first case, they are saying, what is the opposite of edit?
 So they say, they ask it in Punjabi.
 So they know that they want meaning of this word edit and the rest of it, they ask in
 Punjabi and the model says something that the opposite of this is something else.
 And then the person can say, no, I want synonyms.
 And there's like one missing piece here, which is that you have to tell the user, and then
 means opposite in Punjabi.
 So they know what the model is, you know, it's trying to say.
 Okay, so you could interact with this thing sort of across languages and you could prime
 it to say, which parts do I want in which language?
 Because it would obviously not know, I guess, what you want the answer in.
 Yeah, yeah, you can definitely add language tags and that could definitely be it.
 I mean, it's a pretty cool example of exactly of personalization, right?
 Because you can imagine you personalize this exactly to sort of how you want to interact
 with it.
 And someone else who might be more or less skilled at English or in reverse in Punjabi
 might do a different thing.
 That's pretty cool.
 Yeah, there's one more point I wanted to mention which you kind of mentioned earlier with respect
 to the prompt.
 So as you noticed in our prompt, the model does not only give out the answer, it also
 gives out its understanding of the question.
 And I think that's a very kind of crucial piece in this design, because one of the bottlenecks
 for us earlier was the system, a system that is used that the user knows the real answer
 is not really practical because if the user knew the answer, they would be playing with
 the model right outside of an annotation setting.
 So this kind of breaks that barrier.
 So you might not know what the answer is, but you know for sure what you ask for.
 So you can always tell the model, no, this is not what I don't know if you're right,
 but I know for sure this is not what I want.
 And that kind of helps in improving the performance.
 The performance of the model itself might be whatever it is, but we are helping the
 model in understanding that intent more precisely.
 That's the main trick here.
 Yeah I like this getting the answer with the understanding.
 I think that's pretty powerful, not only to interact with the model, but also just to
 understand what it does instead of just getting a simple answer.
 It could be a good recipe for other applications as well.
 Did you have to fiddle around a lot with sort of the prompt structure or the structure of
 what to add?
 Right now you have a bar and then clarification and then colon.
 Is this the first try and it worked or is this the result of many hours of sweat and
 tears?
 No, so it's a first try and we did not impose any intention because our goal was not to
 show our game.
 The goal was to give it words.
 And you know this weird hash and new line.
 This is what we took from OpenAS website.
 They had a bunch of instructions on best practices for formatting your prompt.
 I think they have changed it since, but we just took it from OpenAS website.
 And this was also one of the main motivations like even if I don't know how to exactly have
 the prompts here, there are two ways in which you could gain improvements here.
 One is in context examples within the prompt and the other is at the question side.
 There are like just two aspects for fiddling with this.
 And there has been a lot of work on how to give the right in context examples, what order,
 what examples, how to select them.
 Our focus is on the question part, like only on the input part which comes from the user.
 And we are trying to pull all the knobs, like turn all the knobs at that end and in some
 sense we were able to overcome some limitations which our prompts probably have.
 Maybe there are much better ways of coming up with a prompt than we have.
 But I think all those methods are just, if we plug in any of the nicer methods to come
 up with a better prompt, that's just icing on the cake for us.
 If this was first try and it's still in there, so obviously it worked, was there things that
 didn't work out over the course of this research?
 Like things where you got stuck or maybe even ideas that you had to discard halfway through?
 I can tell one which really bothered us for a long time.
 It's on contrastive prompting, which is we wanted to also give negative answers.
 Can the user just say, no, that's not the right answer.
 With autoregressive models, it is really difficult to somehow give them steer away from probability
 mass towards certain tokens.
 It's really difficult to do that.
 We are still not able to effectively do that.
 Ideally, in the real world, users will give, I think users will give feedback of the kind
 instead of clarifications.
 In addition to clarification, they can also say, no, this is not right or this is why
 it's not right.
 The model came up with what's the capital of India and it says the capital is Mumbai.
 I just want to say, no, it is not.
 It is like Delhi or like you're looking at the wrong places.
 That's something which we were not able to do.
 I think it's an open problem, like this kind of negative prompting.
 It's valuable from a feedback perspective for the future.
 We just don't know how to solve it right now.
 What did you do?
 You played obviously a little bit with these large models with the API, presumably also
 tried out yourself a lot of things I can only assume over the course of this research.
 Is there anything maybe also a bit independent of the research itself?
 Is there anything that you came across that surprised you about these large models and
 how people can interact with them?
 I think for me, one of the things that XB stood out from early days is how good copilot
 was.
 I think if you really have been using it on a day to day basis, and I have been using
 it for a few months now, it has consistently gotten better.
 Initially it had these small weird quirks.
 These models basically generate left to right or top to bottom.
 If I have some, but when you program, you would write some functions below and then
 you go back up to a function and you want to reference the function below.
 So that did not work earlier.
 So it would only condition on things that it had seen so far in the file.
 But they have improved the whole that stuff also.
 So I think it's astonishing that at least in the structure setting, how good they are
 for generating things.
 At the same time, it's also interesting that even when you have 175 billion parameters,
 how poor the model is at common sense, because it's very clear when you go from these structured
 settings to a more open ended setting, the common sense generation or common sense medium,
 I still think the models struggle a lot.
 So it still is clear that there's a long way to go.
 So there's a bit of hope.
 So I think you have to choose your end application wisely.
 But there are clearly very cool applications that can be built for which you don't need
 AGI, as long as you have a very good pattern manager.
 One of the surprises for me was on like just the fact that these models are correctable,
 you know, like a model can make mistakes which are hopeless, you know, it's just total understanding
 is wrong.
 But I think over time, what has happened is with larger models, even though there might
 be many claims that it is missing common sense, and it is, you know, these models are dumb
 and so on.
 But I do believe that, you know, for a certain question, yes, there might be cases where
 it's not coming up with the right answer, but they're still correctable.
 They're not dumb anymore.
 I think these models are getting they're correctable in the sense that their output is not completely
 off and with some guidance, they can get to the right answer.
 Awesome.
 Is there something other than that, that you feel I have maybe not touched in my review
 that you would like viewers to know or, you know, be able to understand or anything that
 I've maybe gotten wrong?
 I think most of the stuff you said was correct.
 Like it was nothing was wrong, really.
 Your understanding and almost everything was was correct.
 Just the only thing I'm not I'm not fishing for compliments.
 Legitimately, if there's something that you feel like, you know, people should know about
 this that we haven't talked about at all.
 Yeah, yeah.
 I think the part about that you mentioned in your video about the feedback could be
 misleading.
 I think we'd be best upon it.
 But I think that's a valid criticism that still holds.
 And that was one of the things that we have not been able to solve even now.
 So we are we are we are trying different kinds of retrieval conditioning on the expected
 output doing something like you said, more complex in one of those four modules.
 But I think that remains a valid criticism of the work that there would be cases where
 feedback would distract.
 So the model was going to say the right thing, but because you have this thing, it's saying
 the wrong thing.
 But we think that problem is kind of there's an easier to solve it is it's to show both
 the answers to the user and let the user pick one.
 So we show this is the answer that I would have given you.
 This is what I would give you with some feedback.
 Pick one.
 But if you don't want to do that, then it's kind of very challenging because the model
 somehow has to know that it's going to make a mistake and only then it's it should pull
 up feedback, etc.
 And those are kind of having it's very hard for models to know that they're wrong or to
 know what they don't know.
 So that's a big challenge and kind of one interesting research direction that we are
 pursuing outside of this, which is how can we let a model know that they don't know or
 then start it going wrong and what can we do in those cases?
 I agree.
 And if you can, if you can do that with a model that you don't even have access to,
 I think that would be a little bit of a little bit of a grail of research.
 That would be seriously cool.
 And I think it would it would improve a lot of applications of these models around, you
 know, all around technology.
 Cool.
 Well, Niket and Aman, thank you very much for being here.
 It was a pleasure.
 And I hope this work goes on and becomes more powerful over time.
 Thanks, Henrik.
 Thank you.
 Thank you so much for having us.
 Thank you.