Memory-assisted prompt editing to improve GPT-3 after deployment (Machine Learning Paper Explained)
#nlp #gpt3 #prompt

Large language models such as GPT-3 have enabled many breakthroughs and new applications recently, but they come with an important downside: Training them is very expensive, and even fine-tuning is often difficult. This paper presents an adaptive method to improve performance of such models after deployment, without ever changing the model itself. This is done by maintaining a memory of interactions and then dynamically adapting new prompts by augmenting them with memory content. This has many applications, from non-intrusive fine-tuning to personalization.

Sponsor: Introduction to Graph Neural Networks Course
https://www.graphneuralnets.com/p/introduction-to-gnns?coupon_code=SUNGLASSES&affcode=999036_lzknae-d

OUTLINE:
0:00 - Intro
0:40 - Sponsor: Introduction to GNNs Course (link in description)
1:30 - Paper Overview: Improve GPT-3 after deployment via user feedback
5:30 - Proposed memory-based architecture
13:00 - A detailed look at the components
15:00 - Example tasks
24:30 - My concerns with the example setup
26:20 - Baselines used for comparison
29:50 - Experimental Results
34:20 - Conclusion & Comments

Paper: https://arxiv.org/abs/2201.06009
Code & Data: https://github.com/madaan/memprompt

Abstract:
Large LMs such as GPT-3 are powerful, but can commit mistakes that are obvious to humans. For example, GPT-3 would mistakenly interpret "What word is similar to good?" to mean a homonym, while the user intended a synonym. Our goal is to effectively correct such errors via user interactions with the system but without retraining, which will be prohibitively costly. We pair GPT-3 with a growing memory of recorded cases where the model misunderstood the user's intents, along with user feedback for clarification. Such a memory allows our system to produce enhanced prompts for any new query based on the user feedback for error correction on similar cases in the past. On four tasks (two lexical tasks, two advanced ethical reasoning tasks), we show how a (simulated) user can interactively teach a deployed GPT-3, substantially increasing its accuracy over the queries with different kinds of misunderstandings by the GPT-3. Our approach is a step towards the low-cost utility enhancement for very large pre-trained LMs. All the code and data is available at this https URL.

Authors: Aman Madaan, Niket Tandon, Peter Clark, Yiming Yang

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, this is a comprehensive paper review on the paper called Memory Assisted Prompt Editing to Improve GPT-3 After Deployment. As the title says, this paper is really cool because it is able to improve these large language models after they're deployed. So this video right here is a comprehensive review on the paper. After you've watched the video, you'll have a good idea of what the method does, what it is, and what the paper describes. The next video released tomorrow will be an interview with the authors of the paper. And that is also really cool. And I definitely learned a lot from that as well. So I invite you to check out both and I'll see you around. Have fun. Hey there, today's sponsor is the course on Introduction to Graph Neural Networks. This is a course by my friend, Zach Jost, who is an expert in graph neural networks. He's packed all his knowledge into one course that will educate you on both the theoretical and hands-on practical aspect on graph neural networks. Graph neural networks are really important. They're definitely one of the most interesting areas in deep learning right now. They've also powered a lot of recent advances in scientific breakthroughs, such as alpha fold protein structure predictions or better traffic predictions. If you use my link, you'll get a 15% discount on the course. Enrollment is open right now and lasts until April 1st or until spaces run out. All right, let's get into the video now. See ya. Hello there. Today, we're looking at Memory Assisted Prompt Editing to Improve GPT-3 After Deployment by Amon Madan, Niket Tandon and others. So this paper proposes a method to improve GPT-3 in an interactive mode, in a user feedback mode. Here is a little sample of how that could look like. So the user would pose a question to GPT-3, for example, what word is similar to good? And this is not displayed here, but in advance of that, there would be like an entire prompt, like you would be used to for prompting GPT-3. If you don't know about GPT-3, I've made a video on GPT-3, extensively describing how that works and how to construct these prompts right here, so that GPT-3 gives you what you want, supposedly, because it doesn't always work. For example, here, the user asks, what word is similar to good? And GPT-3 says, the homonym of good is wood, which is kind of true, but the user is not specified clearly what similar means. The user here had a different intent, which then the user specifies. The user says, similar to means with a similar meaning. So the user didn't mean a word that sounded like good, which is wood. The user meant a word that is kind of like a synonym instead of a homonym. So in this new system, this thing right here would be called feedback, and the user would be able to give this feedback to GPT-3, and then GPT-3 would write that to memory. It's not actually GPT-3. It's sort of like a plugin that the paper develops. And then the user, the next time the user asks, for example, what word is similar to surprised? The system will remember that the last time the user asked a question like that, like similar to, you know, what word is similar to another word, the system will go back to the memory, retrieve the feedback right here, put it into the prompt, and then guides GPT-3 to actually answer in the correct way. And so GPT-3 here says, the synonym of surprised is amazed. So multiple things to see right here. First of all, their plugin, the system that the paper here proposes, can be added to any pre-trained language model, and the language model itself doesn't have to be changed, which is really important for something like GPT-3, because that's too big to change. I guess you can fine tune it, but you'd need a lot more data than just two or three examples. The other thing is that it is interactive. So this is an interactive user session where the user can specify not only clarifications for things that are clearly wrong, but also maybe personal preferences. So this goes beyond what this paper shows. This paper is mostly about either factual accuracy, like accuracy of the task, or figuring out user intent from ambiguous meanings. This could easily be used to personalize interaction with GPT-3 for particular users by interactively letting them improve the system. This is like what normies think of AI, is like a system that learns from the two or three times that I give it feedback, and then gets better over time. So this is pretty cool. Lastly, what was I going to say? I don't remember anymore. But we're going to look at how this works, and what's good about it, what's bad about it. And yeah, that's about it. So here is the proposed before and after of the system. If the user with no memory asks GPT-3, the user gives an X. As we said, it's always prefixed with some sort of a prompt that guides GPT-3 into giving the correct answer structure or type of answer if we're going to look at some of these prompts in just a second. And GPT-3 will give some sort of an answer. Now, this might be good or bad, as you may have seen, it can turn out not in the best way. So in their memory enhanced GPT-3 example, the user would give a question X. Now, let's disregard the memory for now. Let's just go directly to GPT-3, which is what happens in the very first iteration of this interaction. So GPT-3 now has a prompt in front of it as well, but a prompt that the author is here designed, such that GPT-3 doesn't only give the answer to the question, but also you, the understanding of what the user meant. So up here, you can see that by GPT-3 answers, the homonym of good is would, right? GPT-3 doesn't just answer would, which would be the answer, but also this first part right here, which is this understanding. So the authors construct this sort of meta prompt that they give, and that instructs GPT-3 not only to give the answer, but also to give the understanding, like a clear output of what it understood. The user can then take that and decide if that's what the user wanted or not. So if the user is happy, then all is good. If the user is not happy, the user can give feedback to GPT-3. The user gives feedback in natural language, just like types it up, like, no, I didn't mean this, I meant this other thing. And you have to type it up in a bit of a special way. You have to type it up. You can't just say no, I guess you can, but it's best if you write similar to, means with a similar meaning, so you clarify your original question right here. And by doing that, you commit it to the memory. Now, obviously, what you could do is you could simply add that clarification to the prompt, go back to GPT-3 and actually let it answer correctly, which would work. But we're not only about this prompt. The idea here is that this feedback will help guide GPT-3 in all subsequent prompts because the user is likely going to express themselves in the same way. GPT-3, if it misunderstood, is likely going to misunderstand in the same way. So this memory serves as a bit of a generalizable correction mechanism that learns from few items of feedback. So let's look what happens the second time around. So the second time the user again has a question X, we then go first to the memory and we see, or X prime, let's call that X prime. We see, is there anything in the memory that is similar to X prime? Meaning that is there any question before that has been submitted to GPT-3 in the current session? Doesn't need to be in the same prompt or anything, just in the current user session that has been misunderstood. So do we have an instance that is close to X prime where feedback was given? That would be part of the memory. And this is being done with either semantic similarities or so you take some sort of a language model or some sort of a sequence model, for example, a transformer. You look at the embeddings of the sentences, you compare them via cosine similarity. You can also do word overlap or something like this. But what you want to do is you want to retrieve those instances of feedback and then you want to add that feedback to the prompt in the very case, in the case that you... So this is hidden here. This is hidden, it just says, and adds to prompt. And we're going to see how this happens, how the system adds that to the prompt. It's actually quite simple. It's mainly a concatenation, adds it to the prompt. So the users, this is the X prime right here. The X prime is being augmented with the feedback that the user has given previously and then submitted to GPT-3. And with that feedback, GPT-3 is now able to actually more likely give the correct answer. And if it's misunderstood, the user can give feedback again. And that would make it even better in the next few iterations. So this is the overarching system. The paper makes pretty clear that it doesn't propose, like it doesn't purport to be the state of the art or the final system in this framework. It simply wants to present a framework. It states that, I think, two times or more. Now, I have mixed opinions on papers that say, well, we just want to present a framework. On the one hand, it's obviously good to present a framework. Your papers shouldn't be rejected if they have a good idea for a new framework just because they can't get it to be super duper performant. On the other hand, saying, we just want to propose a framework is very often, it's either a cop out for not reaching good numbers or just kind of like, you know, we want to split one paper into two papers because the next paper is going to be sort of the well performing thing. Or it just, there's a danger that it's not super well thought through because the authors haven't actually put in like massive efforts into making this good, at which point many flaws reveal themselves in these types of frameworks. But the framework is pretty general. So, you know, we'll give them that. They claim, yeah, so this is what I just explained. They maintain a memory M of feedback as a set of key value pairs. The key is a misunderstood question and the value is the user's feedback to correct that misunderstanding. Given a new question, we check if the model has made a mistake on a similar question earlier by querying the memory for a similar question, if found, append the corresponding feedback to the question prompt. And here is where they say not definitive, rather our main contribution is the general framework itself, suggesting how user feedback might continuously improve model performance without retraining in a few short prompt setting. So let's look in a little bit more detail into the system, the system has four distinct parts. This memory that we've just talked about, that's a growing table of key value pairs, the key being questions that have been misunderstood and the value being user feedback. So obviously, the user only chooses to give feedback if the user was misunderstood. And therefore, the memory only contains those things. There's a lookup function, which I guess is the most complicated or most complex or complicated, which I'm too surraged. The most complicated of the functions right here, it's they call it a learned retriever that matches the query against all the keys of M. So that's where we retrieve similar prompts that have been misunderstood in the past. And as I said, we can do that with a pre trained embedding, for example, of a transformer model or any any sort of embedding model for text or any other thing. They use Levenstein distance for some experiments. So the combiner is a gating function allowing irrelevant retrieved feedback to be ignored. I don't think they actually do. I don't think they do that right now to ignore irrelevant feedback other than thresholding the lookup function. So the lookup function is an inner product. And I guess the combiner is the threshold on that inner product. The prompter here, it passes the output of the combiner to the prompt. And so that in our case, this is just going to be a concatenation of the prompt and whatever the combiner outputs. So it's going to be the prompt plus the question if there was nothing found in the memory or the prompt plus the question plus the feedback if it was found in memory. So I would add. Yeah, let's let's get into the task and then we'll get into the actual examples. So they have two kinds of tasks. The first kind of tasks, there are five tasks that are broadly in the category of word scrambling and manipulation, for example, to reorder some letters. These are reordered in exact reverse. Other there are other there are anagram one anagram two and so on. There are very various tasks, five of these, and there are five lexical QA tasks which are asking GPT three for a synonym for an antonym for a homonym and so on. They say for each task, the prompt contains a few different variations. For example, what is the homonym of a word? What sounds like the word? They create a data set. So this is where we'll get to that as well. They create a data set of samples, feedback, understanding and the solution. So essentially without the feedback, this would be what you would give to GPT three as a prompt. They also collect feedback so they can simulate users. So they give the X to GPT three. And if it is misunderstood, they do that in a they determine that in a heuristic way. They also provide the feedback to the memory. They come up with sort of invented data of users being understood or misunderstood. The retriever, as I already said, is either a semantic similarity using the cosine distance with a threshold or a lexical similarity and heuristics for similarity matching. The combiner concatenates X and the feedback received by the retriever. And the prompter concatenates the prompt and whatever the combiner outputs. We didn't have one of them, no? Oh, no, the combiner is the gating function. OK, that doesn't it doesn't seem like much of a gating function. Yeah, so I want to jump over the results quite quickly to show you some examples of how that even might look like. So here is a prompt for the tasks. I think these are the lexical the lexical QA tasks. So asking for antonyms and homonyms. This is the entire thing that you would give to GPT three in front of your question. So you would append your question down here somewhere, like below the prompt in the same style as the prompt. So this is this is this is how you query GPT three. What you would do is you would simply simply give some examples and prime GPT three to continue the pattern. So they hear they ask what is the homonym for ring? The homonym for ring is ring. Now, these are all human generated, right? All of these are human generated. So you prime GPT three to, you know, what how how questions are asked and how answers are given. And the important thing right here to see is that all of the answer patterns they provide is it's not just the the answer. For example, permit is the antonym for prohibition. The answer also contains this understanding part. This thing right here, the antonym for prohibition is that's the understanding. And this right here is the label. This is important because the understanding is what the user uses to decide whether or not GPT three has understood the question. What they also do later in the same prompt, they as you can see, they also add questions with feedback. So here you see how they incorporate the feedback. There's like this I don't know what that's called a pipe symbol. And then it says clarification, colon. And then this here is the feedback. So this is also part of the prompt. So the prompt contains some generic feedback where there is some sort of an unclear or ambiguous question. Then there is feedback. And then there is the correct answer that is based on the feedback. So you can see right here, the question is and that's pretty special. The question is or up here, it says, what is the synonym for right? And then the answer is the synonym for is. So it always goes after the question, how the question is formulated. The understanding goes after the question. However, they prime GPT three that if there is a clarification, you can see that the answer. Goes sometimes partially, sometimes fully on the clarification. What I mean by goes on, I mean it. It refers to so the understanding reflects the clarification that allows multiple things. It allows if the user is still not understood, it allows the user to give feedback again. And also it primes GPT three to actually pay attention to this clarification part. So in the prompt, you'll get a bunch of these clarifications to teach GPT three how to include these clarifications in its output. This is pretty smart. It so the prompt is not only a prompt for what kind of answers you want. The prompt is also a prompt for this understanding part, which is a necessary precondition of making the of making the system interactive. And the prompt also includes the next step of the interactivity and how to react to it. This is I think this is a good piece of prompt engineering. People are getting better at this by the day. So this is this is before the question even gets here. So the question would be added here. And if there is feedback in the memory, the feedback would obviously be appended with a pipe symbol and clarification. And then the feedback would be added here. And then GPT three would be prompted to give its answer right here. You can see if there is something in the memory, GPT three already knows how to use these clarification parts right here. So it's pretty good. Yeah, that's there. There are a bunch of examples. You can we can we can maybe look at them or you can look at them. What I want to look at lastly is the data set generation. So they simply say that they created a data set. We manually created 15 task templates with three variants of phrasing the question for each task. You know, this is this is fine. This is prompt engineering. They also they also do come up with sort of the variations for the feedback. Where have I data sets, templates, phrasing each question? OK, I cannot I can't come up with, but it is my understanding that they create the entire data set. So they create the prompts and then the tasks they get from other papers. For example, the synonyms, the homonyms and so on, they get from other data sets that other papers have as well. But then the feedback, the feedback, they also do themselves. And there is a danger right here because they create the task samples for prompting. Right. And also us here. They they create they create the prompts. They create the task samples for the prompts. They also create the example feedbacks and they create the data set of feedbacks, which is dangerous because that might lead to, you know, me just kind of formulating these tasks at templates, not as accurately as, you know, maybe I could. And then obviously, once I clarify, I get an improvement. So the data set creation here, if I understand it correctly, being manual is a big interference, I guess, just from a research standpoint with the researchers interest. Like there's a conflict of interest in making this data set and what you want to get out of the data set. So that is just one concern that I would have right here. The other concern, as you can see, is if you're if you're retrieved clarification from the memory. So this thing here comes from the memory. If that is wrong, like if it's actually not related to the question right here, then things could go bad because GPT-3, given the prompt, is explicitly trained to address whatever is in the clarification in its answer. And that could be not not super duper relevant. It could actually be destructive. So GPT-3 could be completely correct in answering the question. Yet, if the clarification is wrong, it could output a wrong answer. And that's that's not entirely, you know, that's not entirely good. Or maybe maybe I've misunderstood something, because what I can also imagine is that the memory contents are somehow appended to the prompt itself. So the question and the clarification, which and that's what I don't know. And that's what I would like to to ask the authors, because it's not entirely clear to me what they do. They compare two different baselines right here. And it could also be that the baselines implement some of what I just said. So, for example, let's go here. The no mem, that's just GPT-3. Then there is the grow prompt and grow prompt says the prompt is continuously grown with a subset of memory M that can fit within the prompt. So I think this grow prompt thing right here, that's where I have my prompt that we've just seen. And then I would just add like all the entries of M or as many as I could here. And then I would add X. So there would be no clarification over here for X. Never in this grow prompt. It would just be that this portion of memory here grows. And there would always be an X and a clarification or a feedback FB and an X and an FB. So all the things that I've gotten wrong in the past would be appended here as pairs of sample and feedback. And then this is compared to this mem prompt system. That's the system that they have. Now, again, it is not clear to me because tech like is not clear to me if their system simply retrieves the most relevant unit here and appends it here instead of the M. So or maybe the all the relevant units, right? In which case, there would also be no feedback here. Or if their system retrieves the most relevant thing and then appends only the feedback to the X right here, I don't know. Like I don't know. It concatenates C at the end of P and C concatenates X and the feedback retrieved. So I'm pretty sure that it's the second one. It appends. It concatenates the feedback to X. However, here it says they use a cosine distance with a threshold of point nine. There is no mention of like a maximum. Like they retrieve the maximal feedback. It seems like this could result in an entire set of feedbacks. Yeah, but I don't want to go too deep into that. I think I've understood correctly. The danger here is that the green stuff like the grow prompt, the way I understand it, is not like a perfect baseline for what they do because the grow prompt inserts the memory samples as such with the original questions. And their system only inserts the it only inserts the feedback after the question that's currently happening. So either we need a baseline that also adds only feedback right here, but selected in a maybe less smart way, or we need as a baseline, a system that selects the feedback in a smart way, but then then tries to prepend the original question with that feedback in front of X and leave X without feedback or without clarification. So I think, you know, just baseline wise, that is what would be needed. But you can see in their experiments, they show, I guess, convincingly that they are able to improve the accuracy. These are our steps. These are not training steps. These are steps of interaction with the system. So the system is never trained and simply interacted with. And this memory is filled up. You can see, interestingly, at the beginning, everything fails, which is interesting, right? Because one would expect that at least this mem prompt system would remain the same. I guess GPT-3 remains the same. But the mem prompt system also declines. Now, if the retriever is pre-trained and fixed and the threshold is selected well, it should not retrieve any clarifications that have nothing to do with the question. So the performance in my mind shouldn't sink this dramatically, which tells me that the max function is just very important. So they probably mostly get the most relevant feedback if it passes the threshold. And here is what happens, I could guess, if that feedback is irrelevant. So it would actually bias the language model towards giving the wrong answer. And only after a while do I have enough feedback collected that I sort of accurately cover what I would like to ask. Yeah, you can see how this gets, I guess, problematic as soon as your domain of requests to GPT-3 increases. Because there probably doesn't need to be a huge domain before you start to over-correct for things. But then you might also just tighten your threshold. So what do I know? However, regarding correcting things, personalization, I think, might be just a really neat application of this. To just sort of nudge GPT-3 into a personalized interaction with the user. And if it misunderstands there, then I would guess it's more mild than here, where it would just kind of like... It essentially negates an output, essentially says, no, that's wrong. What's also interesting is that the grow prompt never reaches the potential. Again, we don't know if that is because it's a different structured prompt. But at least it's partially due to the fact that it's not smartly selected. It simply appends to whatever is last in the last few things in the memory. Also, interestingly, this mem prompt, where the probability of giving feedback is 0.5, it is kind of bad at the beginning. So here, the probability of getting feedback from the memory is only half. So half the time, the memory would have something, but you're not getting it. This is kind of like an artificial limitation on the system. Just your retriever might be bad and not recognize that there's something there. Interestingly, this also grows to the same performance. And I wonder why wouldn't I expect this to be only half the gains, because it only in half the time, it actually gets any clarification. So half the time, GPT-3 would still output the wrong answer. I might confuse something here, but it seems to me that that's what should happen. They shouldn't end up at almost the same performance. So that is the overview largely over the results. They have these other tasks as well. They're much kind of less clear. They say, well, there's not too many ways to misunderstand in, please turn a word around or so. They also do experiments in low resource languages, which is also cool. Turns out about the same as you can see right here. So in conclusion, I think this is a neat idea. I like that it is essentially a suggestion on how to personalize these language models or how to adjust them, how to make them learn from very, very few things that are nonetheless bigger than prompt. So if you want to teach GPT-3 a new trick and it sort of exceeds the prompt size, this might be a very good way to go if you don't want to go ahead and fine tune it, which would require much, much more data. What I don't really like about this paper is the fact that they say, oh, we just present the framework. It has its good things, but also its bad things. They do actually implement something which is to be commended. But there, I think the sort of comparison with the baseline is shaky because it's not an exact ablation of what they do. There would be better things. And their results, though, are convincing, apart from the fact that I suspect the data set creation was done by the same people who run the study. And since as far as I can understand it, everything except for the actual synonyms of words, everything else was done in a manual fashion, like coming up with prompts, coming up with potential feedback that would warrant at least some caution. Or maybe one would need to look at the exact data set. And as far as I understand it, that is actually available. So we're able to do that. All right. That was it for this paper. Thanks for listening. Let me know what you think of this paper. It seems like a pretty neat idea. And I am excited to see what other people will expand on it. Bye bye.
 Hello, this is a comprehensive paper review on the paper called
 Memory Assisted Prompt Editing to Improve GPT-3 After Deployment.
 As the title says, this paper is really cool because it is able to improve these
 large language models after they're deployed.
 So this video right here is a comprehensive review on the paper.
 After you've watched the video, you'll have a good idea of what the method does,
 what it is, and what the paper describes.
 The next video released tomorrow will be an interview with the authors of the paper.
 And that is also really cool.
 And I definitely learned a lot from that as well.
 So I invite you to check out both and I'll see you around.
 Have fun.
 Hey there, today's sponsor is the course on Introduction to Graph Neural Networks.
 This is a course by my friend, Zach Jost, who is an expert in graph neural networks.
 He's packed all his knowledge into one course that will educate you on both the theoretical
 and hands-on practical aspect on graph neural networks.
 Graph neural networks are really important.
 They're definitely one of the most interesting areas in deep learning right now.
 They've also powered a lot of recent advances in scientific breakthroughs,
 such as alpha fold protein structure predictions or better traffic predictions.
 If you use my link, you'll get a 15% discount on the course.
 Enrollment is open right now and lasts until April 1st or until spaces run out.
 All right, let's get into the video now.
 See ya.
 Hello there.
 Today, we're looking at Memory Assisted Prompt Editing to Improve GPT-3 After Deployment
 by Amon Madan, Niket Tandon and others.
 So this paper proposes a method to improve GPT-3 in an interactive mode, in a user feedback mode.
 Here is a little sample of how that could look like.
 So the user would pose a question to GPT-3, for example, what word is similar to good?
 And this is not displayed here, but in advance of that, there would be like an entire prompt,
 like you would be used to for prompting GPT-3.
 If you don't know about GPT-3, I've made a video on GPT-3, extensively describing how that works
 and how to construct these prompts right here, so that GPT-3 gives you what you want,
 supposedly, because it doesn't always work.
 For example, here, the user asks, what word is similar to good?
 And GPT-3 says, the homonym of good is wood, which is kind of true,
 but the user is not specified clearly what similar means.
 The user here had a different intent, which then the user specifies.
 The user says, similar to means with a similar meaning.
 So the user didn't mean a word that sounded like good, which is wood.
 The user meant a word that is kind of like a synonym instead of a homonym.
 So in this new system, this thing right here would be called feedback,
 and the user would be able to give this feedback to GPT-3,
 and then GPT-3 would write that to memory.
 It's not actually GPT-3.
 It's sort of like a plugin that the paper develops.
 And then the user, the next time the user asks, for example,
 what word is similar to surprised?
 The system will remember that the last time the user asked a question like that,
 like similar to, you know, what word is similar to another word,
 the system will go back to the memory, retrieve the feedback right here,
 put it into the prompt, and then guides GPT-3 to actually answer in the correct way.
 And so GPT-3 here says, the synonym of surprised is amazed.
 So multiple things to see right here.
 First of all, their plugin, the system that the paper here proposes,
 can be added to any pre-trained language model,
 and the language model itself doesn't have to be changed,
 which is really important for something like GPT-3,
 because that's too big to change.
 I guess you can fine tune it,
 but you'd need a lot more data than just two or three examples.
 The other thing is that it is interactive.
 So this is an interactive user session where the user can specify
 not only clarifications for things that are clearly wrong,
 but also maybe personal preferences.
 So this goes beyond what this paper shows.
 This paper is mostly about either factual accuracy,
 like accuracy of the task, or figuring out user intent from ambiguous meanings.
 This could easily be used to personalize interaction with GPT-3
 for particular users by interactively letting them improve the system.
 This is like what normies think of AI,
 is like a system that learns from the two or three times that I give it feedback,
 and then gets better over time.
 So this is pretty cool.
 Lastly, what was I going to say?
 I don't remember anymore.
 But we're going to look at how this works,
 and what's good about it, what's bad about it.
 And yeah, that's about it.
 So here is the proposed before and after of the system.
 If the user with no memory asks GPT-3, the user gives an X.
 As we said, it's always prefixed with some sort of a prompt
 that guides GPT-3 into giving the correct answer structure or type of answer
 if we're going to look at some of these prompts in just a second.
 And GPT-3 will give some sort of an answer.
 Now, this might be good or bad, as you may have seen,
 it can turn out not in the best way.
 So in their memory enhanced GPT-3 example, the user would give a question X.
 Now, let's disregard the memory for now.
 Let's just go directly to GPT-3,
 which is what happens in the very first iteration of this interaction.
 So GPT-3 now has a prompt in front of it as well,
 but a prompt that the author is here designed,
 such that GPT-3 doesn't only give the answer to the question,
 but also you, the understanding of what the user meant.
 So up here, you can see that by GPT-3 answers,
 the homonym of good is would, right?
 GPT-3 doesn't just answer would, which would be the answer,
 but also this first part right here, which is this understanding.
 So the authors construct this sort of meta prompt that they give,
 and that instructs GPT-3 not only to give the answer,
 but also to give the understanding, like a clear output of what it understood.
 The user can then take that and decide if that's what the user wanted or not.
 So if the user is happy, then all is good.
 If the user is not happy, the user can give feedback to GPT-3.
 The user gives feedback in natural language, just like types it up,
 like, no, I didn't mean this, I meant this other thing.
 And you have to type it up in a bit of a special way.
 You have to type it up.
 You can't just say no, I guess you can, but it's best if you write similar to,
 means with a similar meaning, so you clarify your original question right here.
 And by doing that, you commit it to the memory.
 Now, obviously, what you could do is you could simply add that clarification to the prompt,
 go back to GPT-3 and actually let it answer correctly, which would work.
 But we're not only about this prompt.
 The idea here is that this feedback will help guide GPT-3 in all subsequent prompts
 because the user is likely going to express themselves in the same way.
 GPT-3, if it misunderstood, is likely going to misunderstand in the same way.
 So this memory serves as a bit of a generalizable correction mechanism
 that learns from few items of feedback.
 So let's look what happens the second time around.
 So the second time the user again has a question X, we then go first to the memory and we see,
 or X prime, let's call that X prime.
 We see, is there anything in the memory that is similar to X prime?
 Meaning that is there any question before that has been submitted to GPT-3 in the current session?
 Doesn't need to be in the same prompt or anything, just in the current user session that has been misunderstood.
 So do we have an instance that is close to X prime where feedback was given?
 That would be part of the memory.
 And this is being done with either semantic similarities or so you take some sort of a language model
 or some sort of a sequence model, for example, a transformer.
 You look at the embeddings of the sentences, you compare them via cosine similarity.
 You can also do word overlap or something like this.
 But what you want to do is you want to retrieve those instances of feedback
 and then you want to add that feedback to the prompt in the very case, in the case that you...
 So this is hidden here.
 This is hidden, it just says, and adds to prompt.
 And we're going to see how this happens, how the system adds that to the prompt.
 It's actually quite simple.
 It's mainly a concatenation, adds it to the prompt.
 So the users, this is the X prime right here.
 The X prime is being augmented with the feedback that the user has given previously
 and then submitted to GPT-3.
 And with that feedback, GPT-3 is now able to actually more likely give the correct answer.
 And if it's misunderstood, the user can give feedback again.
 And that would make it even better in the next few iterations.
 So this is the overarching system.
 The paper makes pretty clear that it doesn't propose, like it doesn't purport to be the state of the art
 or the final system in this framework.
 It simply wants to present a framework.
 It states that, I think, two times or more.
 Now, I have mixed opinions on papers that say, well, we just want to present a framework.
 On the one hand, it's obviously good to present a framework.
 Your papers shouldn't be rejected if they have a good idea for a new framework
 just because they can't get it to be super duper performant.
 On the other hand, saying, we just want to propose a framework is very often,
 it's either a cop out for not reaching good numbers or just kind of like,
 you know, we want to split one paper into two papers because the next paper is going to be sort of the well performing thing.
 Or it just, there's a danger that it's not super well thought through
 because the authors haven't actually put in like massive efforts into making this good,
 at which point many flaws reveal themselves in these types of frameworks.
 But the framework is pretty general.
 So, you know, we'll give them that.
 They claim, yeah, so this is what I just explained.
 They maintain a memory M of feedback as a set of key value pairs.
 The key is a misunderstood question and the value is the user's feedback to correct that misunderstanding.
 Given a new question, we check if the model has made a mistake on a similar question earlier
 by querying the memory for a similar question, if found, append the corresponding feedback to the question prompt.
 And here is where they say not definitive, rather our main contribution is the general framework itself,
 suggesting how user feedback might continuously improve model performance without retraining in a few short prompt setting.
 So let's look in a little bit more detail into the system, the system has four distinct parts.
 This memory that we've just talked about, that's a growing table of key value pairs,
 the key being questions that have been misunderstood and the value being user feedback.
 So obviously, the user only chooses to give feedback if the user was misunderstood.
 And therefore, the memory only contains those things.
 There's a lookup function, which I guess is the most complicated or most complex or complicated,
 which I'm too surraged.
 The most complicated of the functions right here, it's they call it a learned retriever that matches the query against all the keys of M.
 So that's where we retrieve similar prompts that have been misunderstood in the past.
 And as I said, we can do that with a pre trained embedding, for example, of a transformer model or any any sort of embedding model for text or any other thing.
 They use Levenstein distance for some experiments.
 So the combiner is a gating function allowing irrelevant retrieved feedback to be ignored.
 I don't think they actually do. I don't think they do that right now to ignore irrelevant feedback other than thresholding the lookup function.
 So the lookup function is an inner product.
 And I guess the combiner is the threshold on that inner product.
 The prompter here, it passes the output of the combiner to the prompt.
 And so that in our case, this is just going to be a concatenation of the prompt and whatever the combiner outputs.
 So it's going to be the prompt plus the question if there was nothing found in the memory or the prompt plus the question plus the feedback if it was found in memory.
 So I would add.
 Yeah, let's let's get into the task and then we'll get into the actual examples.
 So they have two kinds of tasks. The first kind of tasks, there are five tasks that are broadly in the category of word scrambling and manipulation, for example, to reorder some letters.
 These are reordered in exact reverse.
 Other there are other there are anagram one anagram two and so on.
 There are very various tasks, five of these, and there are five lexical QA tasks which are asking GPT three for a synonym for an antonym for a homonym and so on.
 They say for each task, the prompt contains a few different variations.
 For example, what is the homonym of a word?
 What sounds like the word?
 They create a data set. So this is where we'll get to that as well.
 They create a data set of samples, feedback, understanding and the solution.
 So essentially without the feedback, this would be what you would give to GPT three as a prompt.
 They also collect feedback so they can simulate users.
 So they give the X to GPT three.
 And if it is misunderstood, they do that in a they determine that in a heuristic way.
 They also provide the feedback to the memory.
 They come up with sort of invented data of users being understood or misunderstood.
 The retriever, as I already said, is either a semantic similarity using the cosine distance with a threshold or a lexical similarity and heuristics for similarity matching.
 The combiner concatenates X and the feedback received by the retriever.
 And the prompter concatenates the prompt and whatever the combiner outputs.
 We didn't have one of them, no?
 Oh, no, the combiner is the gating function.
 OK, that doesn't it doesn't seem like much of a gating function.
 Yeah, so I want to jump over the results quite quickly to show you some examples of how that even might look like.
 So here is a prompt for the tasks.
 I think these are the lexical the lexical QA tasks.
 So asking for antonyms and homonyms.
 This is the entire thing that you would give to GPT three in front of your question.
 So you would append your question down here somewhere, like below the prompt in the same style as the prompt.
 So this is this is this is how you query GPT three.
 What you would do is you would simply simply give some examples and prime GPT three to continue the pattern.
 So they hear they ask what is the homonym for ring?
 The homonym for ring is ring.
 Now, these are all human generated, right?
 All of these are human generated.
 So you prime GPT three to, you know, what how how questions are asked and how answers are given.
 And the important thing right here to see is that all of the answer patterns they provide is it's not just the the answer.
 For example, permit is the antonym for prohibition.
 The answer also contains this understanding part.
 This thing right here, the antonym for prohibition is that's the understanding.
 And this right here is the label.
 This is important because the understanding is what the user uses to decide whether or not GPT three has understood the question.
 What they also do later in the same prompt, they as you can see, they also add questions with feedback.
 So here you see how they incorporate the feedback.
 There's like this I don't know what that's called a pipe symbol.
 And then it says clarification, colon.
 And then this here is the feedback.
 So this is also part of the prompt.
 So the prompt contains some generic feedback where there is some sort of an unclear or ambiguous question.
 Then there is feedback.
 And then there is the correct answer that is based on the feedback.
 So you can see right here, the question is and that's pretty special.
 The question is or up here, it says, what is the synonym for right?
 And then the answer is the synonym for is.
 So it always goes after the question, how the question is formulated.
 The understanding goes after the question.
 However, they prime GPT three that if there is a clarification, you can see that the answer.
 Goes sometimes partially, sometimes fully on the clarification.
 What I mean by goes on, I mean it.
 It refers to so the understanding reflects the clarification that allows multiple things.
 It allows if the user is still not understood, it allows the user to give feedback again.
 And also it primes GPT three to actually pay attention to this clarification part.
 So in the prompt, you'll get a bunch of these clarifications to teach GPT three how to include these clarifications in its output.
 This is pretty smart.
 It so the prompt is not only a prompt for what kind of answers you want.
 The prompt is also a prompt for this understanding part, which is a necessary precondition of making the of making the system interactive.
 And the prompt also includes the next step of the interactivity and how to react to it.
 This is I think this is a good piece of prompt engineering.
 People are getting better at this by the day.
 So this is this is before the question even gets here.
 So the question would be added here.
 And if there is feedback in the memory, the feedback would obviously be appended with a pipe symbol and clarification.
 And then the feedback would be added here.
 And then GPT three would be prompted to give its answer right here.
 You can see if there is something in the memory, GPT three already knows how to use these clarification parts right here.
 So it's pretty good.
 Yeah, that's there.
 There are a bunch of examples.
 You can we can we can maybe look at them or you can look at them.
 What I want to look at lastly is the data set generation.
 So they simply say that they created a data set.
 We manually created 15 task templates with three variants of phrasing the question for each task.
 You know, this is this is fine.
 This is prompt engineering.
 They also they also do come up with sort of the variations for the feedback.
 Where have I data sets, templates, phrasing each question?
 OK, I cannot I can't come up with, but it is my understanding that they create the entire data set.
 So they create the prompts and then the tasks they get from other papers.
 For example, the synonyms, the homonyms and so on, they get from other data sets that other papers have as well.
 But then the feedback, the feedback, they also do themselves.
 And there is a danger right here because they create the task samples for prompting.
 Right. And also us here.
 They they create they create the prompts.
 They create the task samples for the prompts.
 They also create the example feedbacks and they create the data set of feedbacks,
 which is dangerous because that might lead to, you know,
 me just kind of formulating these tasks at templates, not as accurately as, you know, maybe I could.
 And then obviously, once I clarify, I get an improvement.
 So the data set creation here, if I understand it correctly, being manual is a big interference,
 I guess, just from a research standpoint with the researchers interest.
 Like there's a conflict of interest in making this data set and what you want to get out of the data set.
 So that is just one concern that I would have right here.
 The other concern, as you can see, is if you're if you're retrieved clarification from the memory.
 So this thing here comes from the memory.
 If that is wrong, like if it's actually not related to the question right here,
 then things could go bad because GPT-3, given the prompt, is explicitly trained to address whatever is in the clarification in its answer.
 And that could be not not super duper relevant.
 It could actually be destructive.
 So GPT-3 could be completely correct in answering the question.
 Yet, if the clarification is wrong, it could output a wrong answer.
 And that's that's not entirely, you know, that's not entirely good.
 Or maybe maybe I've misunderstood something, because what I can also imagine is that the memory contents are somehow appended to the prompt itself.
 So the question and the clarification, which and that's what I don't know.
 And that's what I would like to to ask the authors, because it's not entirely clear to me what they do.
 They compare two different baselines right here.
 And it could also be that the baselines implement some of what I just said.
 So, for example, let's go here.
 The no mem, that's just GPT-3.
 Then there is the grow prompt and grow prompt says the prompt is continuously grown with a subset of memory M that can fit within the prompt.
 So I think this grow prompt thing right here, that's where I have my prompt that we've just seen.
 And then I would just add like all the entries of M or as many as I could here.
 And then I would add X.
 So there would be no clarification over here for X.
 Never in this grow prompt.
 It would just be that this portion of memory here grows.
 And there would always be an X and a clarification or a feedback FB and an X and an FB.
 So all the things that I've gotten wrong in the past would be appended here as pairs of sample and feedback.
 And then this is compared to this mem prompt system.
 That's the system that they have.
 Now, again, it is not clear to me because tech like is not clear to me if their system simply retrieves the most relevant unit here and appends it here instead of the M.
 So or maybe the all the relevant units, right?
 In which case, there would also be no feedback here.
 Or if their system retrieves the most relevant thing and then appends only the feedback to the X right here, I don't know.
 Like I don't know.
 It concatenates C at the end of P and C concatenates X and the feedback retrieved.
 So I'm pretty sure that it's the second one.
 It appends.
 It concatenates the feedback to X.
 However, here it says they use a cosine distance with a threshold of point nine.
 There is no mention of like a maximum.
 Like they retrieve the maximal feedback.
 It seems like this could result in an entire set of feedbacks.
 Yeah, but I don't want to go too deep into that.
 I think I've understood correctly.
 The danger here is that the green stuff like the grow prompt, the way I understand it, is not like a perfect baseline for what they do because the grow prompt inserts the memory samples as such with the original questions.
 And their system only inserts the it only inserts the feedback after the question that's currently happening.
 So either we need a baseline that also adds only feedback right here, but selected in a maybe less smart way, or we need as a baseline, a system that selects the feedback in a smart way, but then then tries to prepend the original question with that feedback in front of X and leave X without feedback or without clarification.
 So I think, you know, just baseline wise, that is what would be needed.
 But you can see in their experiments, they show, I guess, convincingly that they are able to improve the accuracy.
 These are our steps. These are not training steps.
 These are steps of interaction with the system.
 So the system is never trained and simply interacted with.
 And this memory is filled up.
 You can see, interestingly, at the beginning, everything fails, which is interesting, right?
 Because one would expect that at least this mem prompt system would remain the same.
 I guess GPT-3 remains the same.
 But the mem prompt system also declines.
 Now, if the retriever is pre-trained and fixed and the threshold is selected well,
 it should not retrieve any clarifications that have nothing to do with the question.
 So the performance in my mind shouldn't sink this dramatically, which tells me that the max function is just very important.
 So they probably mostly get the most relevant feedback if it passes the threshold.
 And here is what happens, I could guess, if that feedback is irrelevant.
 So it would actually bias the language model towards giving the wrong answer.
 And only after a while do I have enough feedback collected that I sort of accurately cover what I would like to ask.
 Yeah, you can see how this gets, I guess, problematic as soon as your domain of requests to GPT-3 increases.
 Because there probably doesn't need to be a huge domain before you start to over-correct for things.
 But then you might also just tighten your threshold.
 So what do I know?
 However, regarding correcting things, personalization, I think, might be just a really neat application of this.
 To just sort of nudge GPT-3 into a personalized interaction with the user.
 And if it misunderstands there, then I would guess it's more mild than here, where it would just kind of like...
 It essentially negates an output, essentially says, no, that's wrong.
 What's also interesting is that the grow prompt never reaches the potential.
 Again, we don't know if that is because it's a different structured prompt.
 But at least it's partially due to the fact that it's not smartly selected.
 It simply appends to whatever is last in the last few things in the memory.
 Also, interestingly, this mem prompt, where the probability of giving feedback is 0.5, it is kind of bad at the beginning.
 So here, the probability of getting feedback from the memory is only half.
 So half the time, the memory would have something, but you're not getting it.
 This is kind of like an artificial limitation on the system.
 Just your retriever might be bad and not recognize that there's something there.
 Interestingly, this also grows to the same performance.
 And I wonder why wouldn't I expect this to be only half the gains,
 because it only in half the time, it actually gets any clarification.
 So half the time, GPT-3 would still output the wrong answer.
 I might confuse something here, but it seems to me that that's what should happen.
 They shouldn't end up at almost the same performance.
 So that is the overview largely over the results.
 They have these other tasks as well.
 They're much kind of less clear.
 They say, well, there's not too many ways to misunderstand in, please turn a word around or so.
 They also do experiments in low resource languages, which is also cool.
 Turns out about the same as you can see right here.
 So in conclusion, I think this is a neat idea.
 I like that it is essentially a suggestion on how to personalize these language models or how to adjust them,
 how to make them learn from very, very few things that are nonetheless bigger than prompt.
 So if you want to teach GPT-3 a new trick and it sort of exceeds the prompt size,
 this might be a very good way to go if you don't want to go ahead and fine tune it, which would require much, much more data.
 What I don't really like about this paper is the fact that they say, oh, we just present the framework.
 It has its good things, but also its bad things.
 They do actually implement something which is to be commended.
 But there, I think the sort of comparison with the baseline is shaky because it's not an exact ablation of what they do.
 There would be better things.
 And their results, though, are convincing, apart from the fact that I suspect the data set creation was done by the same people who run the study.
 And since as far as I can understand it, everything except for the actual synonyms of words,
 everything else was done in a manual fashion, like coming up with prompts, coming up with potential feedback that would warrant at least some caution.
 Or maybe one would need to look at the exact data set.
 And as far as I understand it, that is actually available.
 So we're able to do that.
 All right.
 That was it for this paper.
 Thanks for listening.
 Let me know what you think of this paper.
 It seems like a pretty neat idea.
 And I am excited to see what other people will expand on it.
 Bye bye.