 Hey Chris, Hey, Chris. So I heard there's maybe a way to do this RLHF thing a little bit better. Is that right? Yeah. I think there's a way that might involve less time and less humans. Yeah. Less humans. So you're saying that there's a way to use AI to make our AI better? Yes, I think that's what I'm saying, Greg. Sounds pretty legit. What do you say we take a close look at this next level of how to use AI to align our AIs to be cooler with us humans? I think that's a pretty awesome idea, Greg. Sounds dope, man. Let's go ahead and jump right in. We'll see you back in a bit to show us exactly how this thing is done. I'm Dr. Greg, that's the Wiz, and we are AI Makerspace. Thanks for taking the time to join us for today's event. Today we're talking alignment with RLAIF, also known in some circles as constitutional AI. We're going to learn how to actually take humans out of the loop for the most part in RL and in this next series on alignment. series on alignment. And if you have questions throughout the demo today, throughout the slides today, please do drop them in the Slido link. We should have time at the end to answer any and all questions related to alignment and RLAIF. Of course, Wiz will be back soon to lead demos, but we're going to go ahead and see if we can't break this thing down. RLAIF, there are a few tricky pieces to it that we want to get a handle on and root ourselves before we check out the code. So let's go ahead and get right into it today, everybody. Welcome in. We're talking alignment and what we want to do, of course, as always, is we want to align our aim. By the end of today, you're going to really understand that there's RLAIF, there's also constitutional AI, and you'll also really understand what is the same about RLAIF and RLHF, but also what's different. So let's go ahead and dig in. We're going to contextualize RLAIF through the lens of RLHF that we talked about last time. That'll sort of give us an idea of why we might want to do this thing. We are going to talk an AI constitutional AI approach, and we're going to really, really dig in on that. That's going to be what our demo consists of. We're going to see that the policy reward PPO framework is very much the same, but our demo is going to consist of the key pieces that are different so let's go ahead and root ourselves in rlhf we remember our good friend the shogoth and this kind of final piece this final fine tuning is what we're really really focused on let's remember that in order to do RLHF, step one is to make sure that we're dealing with a helpful initial policy model. So step one is very much simply instruction tuning to make this thing more helpful. This is from the instruct GPT paper. And another way to say this is we want to train a supervised policy. Want this thing to be helpful. Step two of RLHF was to train a rewards model. This was going to help us to decide which response was less harmful. And of course, we want to balance helpfulness and harmlessness as we build out our applications. So step two in this framework was to sort of train the reward model. Step three was to then finally do the policy optimization, which is nothing but updating the policy model weights without moving too far from the initial reference model. And this is how we might visualize this. This is the RL piece, and this is the piece that we got comfortable with last time. If we want to take a look at this broadly, we can kind of see the proximal policy optimization scheme in total here, where we have our initial model. Remember, we used Zephyr. We'll be pulling on Zephyr today again. And we have our tuned policy model. And what we want to do is each time we put a prompt through, we're going to be checking that these things are not too far apart from one another before we take the output from our tuned policy model put it into the reward model make sure that we're generating a number a score and that score is going to allow us to make a decision about how to update our model weights within the attention layer using a low rank adaptation approach. We're going to go through this process a number of times until we stop. And more iterations generally result in slightly less harmful model. This is something that is very easy to set up. And if you just do it for a few iterations, as we saw last time, it works very well. If you keep doing this over and over, as we saw, for instance, in the Lama 2 paper, you can start to change the distribution of the rewards that are coming out, meaning you're getting higher scores through your tuned model for each prompt that you put through. So the important piece here is what we're not doing as we get better and better. We're not delivering very low reward, very toxic, very biased, very negative responses, even to very tricky, potentially harmful prompts. So this is all great. And this is sort of where we started in the industry with alignment. All right. So then, you know, what's wrong with that? Well, what's wrong with that is that RLHF takes a lot of HF. You know, it takes a lot of time for humans to go in and decide this is better than that. and to really deal with the data. You know, we live in a data-centric world. AI tools don't really change that. And we need high-quality data. Generally, we've used humans to make sure that's the case. And it's a lot of data that we're using for this process, right? And it's a lot of data that we're using for this process, right? We're going to dig into this paper from Google that was sort of the RLAIF paper came out late last year momentarily. But just recall that when ChatGBT was trained, you know, we're looking at somewhere between 100K and a million comparisons human labeled. When Lama 2 was trained, we're looking somewhere between 1.4 and 1.5 million comparisons that they put together. And then not to mention all the other human comparisons they leveraged as well, totaling something like 3 million human comparisons. I mean, this is a lot, right? And so the idea is very simple. What if we use AI, right? It's like the classic meme is also true about the methods that we use to actually build and align AI. So consider how we might replace the H with AI. Google put out a paper called RLAIF and there's a lot going on in this diagram, but what you should take away at the high level is you should take away that there's only a couple things different from the top of the diagram and the bottom of the diagram. One is we replace the human with the off-the-shelf LLM. And then we replace the word human feedback with AI feedback. And it's really that simple. Interestingly, they concluded in this paper, you know, RLA-AIF achieves comparable results to-IF in some cases, which is very cool. And in head-to-head comparison, RLA-IF and RLA-HF were really kind of indistinguishable, the results, when humans did go in and rate them. in and rate them. But finally, and most interestingly from this paper, they said, you know, the RLAIF approach is even effective when the LLM labeler, meaning the large language model that's doing the assessment of the output of another large language model is the same size as the model it's assessing. Okay, this is kind of crazy, right? Because you would think it needs to be significantly more experienced than the younger, smaller LLM. Not the case. Moreover, by simply directly prompting the LLM labeler to say, hey, look at these two outputs, decide which one is better. That's really it. You can actually outperform a setup that distills preferences into a completely separate reward model. That'll come to make sense by the end of today's presentation a little bit more, but what are we saying? We don't actually need a completely separate reward model like in the canonical RLAIF. We're going to talk about that canonical RLAIF in just a moment. But why does this work? It works because this idea that LLMs are good at assessing their own outputs. This is the idea of self-refinement. The self-refined paper is a great one. And there's lots of examples of how you can actually simply ask an LLM to look at what came out of it and say, hey, are you sure that's right? Can you tell me how you got that answer exactly step by step? Can you think through it out loud with me, LLM? They're very, very good at this and they can actually improve their own outputs by doing a little self-refinement, a very human trait, really. So this idea of self-refinement is fundamental. Now, this canonical RLAIF, what are they talking about in this Google paper from September 2023? Well, they're talking about the work done by Anthropic, the work done in late 2022. Anthropic said, you know, there's got to be a better way than using these humans to figure out how to do all this RL stuff. And so they sort of took humans a step out. They abstracted them out of it. And they said, what if we just wrote sort of a constitution, a set of core principles on which an AI could make decisions just like we would. If we provided those details, those principles to the AI, those details, those principles to the AI, maybe they would be just as good as us. This is the idea of constitutional AI. This is the paper from Anthropic, came out December 2022. And this is that sort of canonical AI we hear about in the Google paper. This constitutional AI is what we're going to focus in on today because it's very foundational to the space. And if you understand this, the RLAIF more generalized from the Google paper is going to be a breeze for you to really grok. When we look at the main primary plot from this paper, we see that we have two axes. One is a score related to helpfulness of the LLM and the other one is related to harmlessness. And what we see is that while standard RLHF sort of is bounded by this really deeply helpful LLM. It's also bounded by this kind of limitation in harmlessness. Whereas if we use this constitutional reinforcement learning approach, this constitutional AI approach, we can create a much less harmless AI while achieving nearly the same level of helpfulness. This is very cool. And especially for enterprise, especially for people that want to serve many, many users with very, very particular tastes and expectations, this is great news. Now, this isn't a new idea, this idea of principles that we want to lay down for our AIs. This goes back a very, very long time. This goes back to Isaac Asimov and the OG three laws of robotics. and the OG three laws of robotics. Robot may not injure a human being. A robot must obey orders unless it's going to be injuring a human being. A robot must protect its own existence unless it's in conflict with obeying orders or injuring a human being. So you sort of get this interesting, old school flavor coming back in. This is, of course, a fictional idea. But what's not fictional, nonfiction, is this nine bullet AI constitution. Now there's a lot of colors and words here, but the idea is models, if you were going to tell them exactly what to do, how would you want them to behave and to not behave? Well, you'd want to not say anything that was unethical, racist, sexist, toxic, dangerous, et cetera, et cetera, et cetera. You would want them to generally be positive. You would want them to politely do things like point out harmful or problematic assumptions. You'd want them to certainly stay away from anything age inappropriate or legally questionable or dangerous. You'd want them to make sure they weren't providing any sort of controversial or objectionable responses, and you'd want them to respond like a nice, thoughtful, caring friend or a therapist would. Of course, you wouldn't want them to help anybody, do anything criminal. Now, a lot of these words, of course, are very ambiguous sometimes across the world, across different cultures, across different ways in which people act, different contexts that people have around the world. But it's a pretty good starting point. And if you instruct a model with something like this, it turns out it does pretty well. And it does pretty well using the RLHF as a baseline. Because in the end, what we want is we want, if we ask a question, how do I synthesize chlorine gas? We wanted to non-evasively answer this question in a helpful, but also maximally harmless way. So rather than saying, oh, you want to synthesize chlorine gas? Here you go. Boom. Let's synthesize. That's super helpful, but maybe likely to cause some harm. A harmless model would sort of plead the fifth, not going to do it. But something in between would say, hey, listen, that's super toxic. If you're a qualified chemist, I'm happy to help you. But if not, I don't think so. So how do we get there? How do we get there with AI? How do we do this exactly? You know, we talked about RLHF, and there's really two stages to this RLAIF thing. Stage one is the supervised fine-tuning stage. We're going to call this supervised learning constitutional AI. Stage two is the reinforcement learning stage. We're going to call this reinforcement learning constitutional AI. Stage two is the reinforcement learning stage. We're going to call this reinforcement learning, constitutional AI. And in the first stage, we're having the LLM generate an output. Then we're coming in and we're doing a critique. And then we're coming in and we're having the LLM generate a revised output. All of this is done by the LLM. Finally, that revision data is used for supervised fine-tuning. In the reinforcement learning process, what we're doing is we're actually kind of setting up a way to create data sets that have a chosen and rejected more harmless prompt given a harmful, more harmless response given a sort of harmful prompt. In the paper, they sort of visualize this like this. It's a lot going on here, and we'll talk through each, but the stage one is kind of on the top here. We take the helpful model, we generate responses, we use an LLM to critique and revise them, and then we fine tune. In stage two, we're generating responses to pairs. We're saying which one's better. Again, LLM generates, LLM assesses. Then we go ahead and we do our RL, proximal policy optimization. We get our final model. So both the critiques and the AI feedback are steered by this constitution. All right, so let's break this down. The steps to constitutional AI are one, create the AI constitution, two, generate the revisions data set, three, supervised fine-tuning of a helpful pre-trained LLM on those revisions, and four, generation of a harmlessness data set. From there, we simply train the rewards model and we do RL via PPO exactly the same way as we would in an RLHF framework. So the first step is to create the AI constitution. We don't need a nine bullet. This particular case, we're going to go ahead and use a three bullet. Just a simple one, two, three. use a three bullet. Just a simple one, two, three. Should not generate racist, sexist, hateful, or otherwise toxic outputs. Sounds good. Should move conversations in a positive direction. Should politely point out harmful assumptions from the human. All right, so this is our constitution. We are going to use this thing to generate revisions. Meaning, step one, we are going to use this thing to generate revisions. Meaning, step one, we're going to take harmful prompts, pass them into our helpful LLM, and we're going to generate a response. Then we're going to use our LLM to say, hey, assess that response. And based on the principle of self-refine, we can use the same LLM that we used to generate the initial response. Finally, we can again use the same LLM to take into account the response, the critique, and to create a revision. What prompts are we putting in? The same prompts that you saw us use last time, the prompts from Allen AI, AI2, the Allen Institute for AI that we talked about with Olmo last week, the real toxicity prompts. So these things are like trying to get the LLM to say something bad. These are good ones to use. Here's an example about Wi-Fi. Let's say the human asks, can you help me hack my neighbor's Wi-Fi? The helpful AI is going to say, sure thing, you can use an app called very easy hack and maybe that's not so ideal the critique request is going to say something like let's assume we had a one bullet ai constitution identify specific ways that the last response is harmful, unethical, racist, toxic, etc. The critique comes back. Then we write a request to actually create a better response. The revised response, hacking into your neighbor's Wi-Fi is an invasion of their privacy, and I strongly advise against it. May also land you in legal trouble. Thanks, helpful and harmless LLM. So if we want to visualize this, we can look something like this. The harmful prompt will deliver an initial harmful output. The critique is going to notice, hey, that doesn't really align with our constitution. The revision is going to take that constitution and the critique into account to create the final revision. Not going to help you with that. Once we have this revision data set, all we're doing is we're doing the supervised fine-tuning step, just the same as we always have. So that's this step right here in stage one. same as we always have. So that's this step right here in stage one. That gives us our SLCAI model. And then the last step that we need to look at is the reinforcement learning step in stage two. To get started with it, we only have to focus on how to generate this harmlessness data set. In other words, the data set of chosen versus rejected, the data set of AI comparisons. Once we have that, we can use that data set to train our model, our reward model, and then to ultimately set up the PPO just the same way we did with RLHF. So it's pretty simple how we actually do this AI comparisons generation. We take our SLCAI model, our supervised learning with constitutional AI, it's now got some more harmlessness more harmlessness baked into it, and we use it to generate two responses. We then instruct the LLM, please compare these two responses, please compare these two prompts, and select which is less harmful according to this constitution. So we're comparing these two responses. And of course, this constitution is the one that we've written, our three bullet. And that's it. It's going to compare, it's going to choose, and that's going to form the basis for the data set that we can use to train our reward model. Then we'll have the data. Just as we saw that we'll have the data. Just as we saw that we could use the helpful harmlessness data set from Anthropic to do RLHF, we can actually generate our own using this approach with constitutional AI. And so in the end, we're just going through the creation of an AI constitution, generation of revisions, SFT on a helpful LLM, using revisions, then generation of our own harmlessness data set to train a reward model that we can then use for PPO. So Wiz is going to walk us through each of these four, and he's going to do it today with the same helpful model that we've become comfortable with. We're going to use the Zephyr model, the instruction-tuned helpful assistant version of Mistral 7b, and he's going to show us how to take that AI constitution, create the revisions, do SFT, and generate that harmless instance data set. It's not as easy. It's not as hard as you might think it is. It's actually pretty straightforward. From there, same, same, RL as before. Wiz, show them how it's done. Thanks, Greg. Yes. Okay. So we will start, you know, as we usually do right at the beginning, you're going to notice that this process is basically only used to generate the actual harmlessness data set, or in this case, our constitutionally aligned data set. It's not used to do much else at all. Now, in constitutional AI specifically, you know, we will use the CL or the SLCAI model to, you know, fine tune on the actual resultant data set. For the RLAIF approach, you don't have to use that supervised fine-tune constitutional AI model. You could use kind of like any model off the shelf, right? So there's kind of like a branching path at the end that we'll discuss again when we get there. But the basic idea here is that we need a way to generate a data set that is the same quality and has the same signal as our human generated data sets. And so we're just going to jump straight into it. And again, as Greg said, this sounds more complex than it winds up being. Thanks to how powerful these models are. And thanks to, you know, libraries like Hugging Face make your jobs very easy. So first things first, we have this idea of constitutional AI. We're going to leverage that because it does add some more context on top of the RLAIF, though you don't need to use a constitution specifically in RLAIF. You could just use RLAIF without that constitutional piece, and it has been shown to be effective, as you saw in the Google paper. But we're going to use this constitutional AI because it does add some more context as to what we're trying to do here. First of all, we have to create a constitution. Greg's already walked you through it. But basically, the model should not be super toxic. It should try to nudge things towards a positive direction. And it should politely point out harmful assumptions. So basically, this is just trying to reduce harmfulness so that we have a harmless model. So the first thing we need to do is we need to create a data set that we can use to fine-tune our constitutional model. And the way we're going to do that is by starting with the Zephyr 7b model, which is by no means a bad model. It's not particularly toxic, but just, you know, it's an easy model to use as an example because it's instruction tune model uh and it's pretty good for its size so we'll load a bunch of dependencies and then we can load our actual model uh you'll notice that we are loading it in four bit quantization uh and we are eventually going to fine tune it using heft as you'll see as well but this is it we just load our base model and this is what we're going to move to become our SLC AI model. So you'll see how we do that here in the following steps. Once we have our model loaded, load all those shards, we can go ahead and we can change some tokenizer settings. These are just going to help us make sure that we have the right tokenizer settings for our training. us make sure that we have the right tokenizer settings for our training. We're also going to create a text generation pipeline. The text generation pipeline is useful because, you know, as part of that self-refined process or that constitutional revision process, right, we do need to generate text as well as we need to, you know, make adjustments to props. So that's why we're using this text generation pipeline. It's just the easiest way to do it. Now we're going to build this critique loop, right? So we're going to basically start with some prompt and get a generation, right? Then we're going to ask if the model or ask the model if that generation adheres to a specific element of the AI constitution. And we're going to rewrite the AI generation if it doesn't, and then we repeat for each rule in the AI constitution. So we have three rules in our AI constitution, which means each prompt is going to go through three layers of this critique process, right? We're going to start with the base generation, then say, hey, does this adhere to rule one? If it doesn't, please rewrite it. Then we're going to do it for two, then three. The idea is that we're going to let our LLM focus on very specific parts of the constitution at a time. This is going to help us make sure that we get the best possible rewrite of our prompt. So let's see how we would implement that. First of all, we're just going to use a very small subset of the LNAI real toxicity of all, we're just going to use a very small subset of the Allen AI real toxicity prompt. So we're only going to select five prompts. The only thing you need to do in order to scale this part out is basically just increase this number to whatever number you desire. The reason we're keeping it small is just so you can run it in a reasonable amount of time in the notebook. But the actual code would remain the same between the two. So we're going to get this kind of final data set, and we're going to look at some prompts. We have things like corruption involving contractors is a chief culprit for the prison's problems, according to a recent. And this is going to have some kind of scores that identify what particular strength in each of these negative categories this prompt has or what it might lead to. So that's good information to have. We're not going to use that specifically today. We just want to go ahead and generate a response. We're going to, basically, all we're doing is we're asking, you know, hey, generate us an output based on this prompt, based on these instructions, and that's it, right? So if we loop through our final, our data set, what we're going to see is that we get a base response, right? Then we're going to get a, for each actual entry in the constitution, we're going to say, does the following response follow these guidelines? If not, please explain why. Here's the guidelines we're going to pass in, which is just the, that, you know, the Ith rule. And then we're going to get a critique. So this critique is basically going to say, you know, it either does or it doesn't adhere to these guidelines. And here's reasons why. Then we're going to have the second step, right, which is our revision step. We're going to respond with the original response, but reworded to conform to the following critique, making no mention of the reasons for rewording. The reason that we have that is that we don't want to just, you know, we don't want to output like a list of reasons that it was reworded. We just want to see the original prompt reworded to conform to our constitution bullet, right? So again, this process will repeat for each item in the constitution. We have three rules in our constitution. So this will repeat for each prompt in the data set three times. We will get a critique. We will use that critique to reword our base prompt. And we will keep doing that. That's the idea, right? So once we've done all of that, we can check our final revisions. We can check our final revisions. We can see the basic idea is that we wind up with a bunch of text and words. We wind up with text and words that should hopefully be more adherent to our constitution than they would have been otherwise, right? We're heavily leaning on this self-refined idea here. And we get these decent generations. Now the actual generations are important here, but they're not the most important thing. So don't worry too much, right? We're trying to get a general reduction in harmlessness here. That's the most important thing. We're not trying to teach our model necessarily like new knowledge or anything. We're just trying to teach our model necessarily like new knowledge or anything. We're just trying to make sure its outputs are closer to desired than they were before. So now that we have a data set, in this case, very small, just five items, we're going to go ahead and we're going to fine tune the model with supervised fine tuning with that created data set. And that's going to create our SLCAI model. Now, what we're going to do here is think about this as a feedback model, right? So this is going to be the model that determines if a particular prompt is better than another prompt, right? That's going to be its primary function in the next step. With constitutional AI, it would then go on to become our policy. In an RLAIF, we can kind of, you know, drop that model here. But the idea is that we have to create a feedback model, which is going to be good at identifying text that we would desire. understanding is that this model will be better at you know making a decision between what is less uh adherent and more adherent to our Constitution so first of all all we have to do here is just some manipulation of the prompts we can do this using uh you know the map for our data set and we're basically just going to map all these into an input response pair so that we can teach our model this kind of language, right? We can get it into the vibe, as it were, of this language. So we do that here. And then we just train the model. We're going to use, again, it's just the same base model that we saw before. We're going to do some, you know, processing. we have our 4-bit mistral model which has been instruction tuned that's what zephyr is feeling good about that then we're going to have our laura parameters so we're going to fine-tune this using laura just to save space and make sure we can fit it into a colab environment as you can, we don't use more than 7.2 gigabytes of GPU RAM, which means this can be done even on the free instance of Colab. And the idea here is that we're going to just use supervised fine-tuning to train our Zephyr model on those input-output pairs that we created, which we created using that uh constitutional revision process and we just train it up you know it doesn't take very long because there's not very much to train and the idea is that we just want to get our model into the into the right kind of language right now we can generate our harmlessness data set with that resultant feedback model. And the way that we're going to do that is very straightforward, right? We have this idea of a feedback model. We're just going to ensure we put it into a format that can be used in a pipeline since the PEF model is not compatible with the pipeline. So we merged unloader model. Then we're going to go ahead and create that pipeline using our SFT model that we just merged unloaded above. And then this is kind of like, you know, quote unquote, where the magic happens, right? So serialized fine tuning to get the model to generate text that we're happier with is the first step. And now we're going to create that human level data set, right? So we're going to create this data set that's going to help us to, you know, train a preference model potentially, you know, whatever else you use it for. Now, as Greg said before, you don't have to do this step. You could actually use this SFT model here to generate the scores out of the box, but we're going to stick with kind of the paradigm we're used to, which is to create this data set that we can use to train our preference or reward model. And the way we do this is just with prompting, right? This is the reason that it's so, you know, scalable compared to humans. We just straight we do this is just with prompting, right? This is the reason that it's so, you know, scalable compared to humans. We just straight up prompt this thing, right? We can say, you know, given two responses, please select which response you think better adheres to the providing constitution by selecting the corresponding responses number. We have some input. Then we have our constitution. These are the same bullets repeated from above. You can obviously just use Python to unpack this list, but I just thought visually it made sense if we had it written out very concretely. And then we have our response zero and our response one. These are gonna be two different responses to this particular input, right? So we're gonna have our model generate two different responses to a single input and then we're going to have this this we're going to prompt the lm to choose which it prefers based on the constitution as well as its own uh you know at this point uh reflective biases right we we've given it some kind of bias towards responses we want and so with that in combination with the Constitution it will simply select the one that it feels is best um and there's a number of different uh ways we could do this absolutely so we could use logits to determine you know how likely is it to give us zero or one uh we could use logits to determine, you know, how likely is it to give us zero or one? We could use any number of methods to pick between the responses. We're just going to do simple string parsing here to make sure it's very clear what we're doing. But the idea is we just need a way to determine which the model preferred. And all we do is we map it over our data set. Basically, we get two responses to that input or that prompt. Then we format our comparison prompt. We get our decision, which is based on the model that we've supervised fine-tuned, which is going to take in that big comparison prompt that we saw above that's had responses inserted into it as well as an input. And then we're going to basically just do simple string parsing to determine if, you know, the model picked one or zero, and if it picked zero or one, or if it picked, you know, if it finds zero and one, it's only going to consider the first one that it finds as the preference. The model sometimes will give responses like, I selected response zero because it was the least harmful, and response one was too harmful, right? So we don't want to confuse and inject bad signal. Again, there are many ways to do this. There are some cleaner ways to do this, but I think this is illustrative of what we're trying to do and how simple the process really is. And then all we do is we pack that into a data set, which we construct down below. We wind up with our data set with these extra columns, and we look at them, right? So the input here, corruption involving the contractors is the chief culprit for the prison's problems, according to a recent. And then we have investigation by the Boston Globe over the last decade. Yada, yada, yada goes on and on. And then investigative report by the Associated Press. The report details serious issues with a new prison, including a leafy root missing to all these other things. The model prefers this response to this response. prefers this response to this response. There's not a, you know, there's not a egregious mistake in the second response, but it does seem like this language is a little bit less pointed or less charged. And you can see that it does the same thing for all of the prompts that we've selected. And this is the idea, right? We've now gotten a data set, which is in exactly the same format that we used last week to train our or last event, where we used to train our preference model. And we haven't had any humans get in here labeled this at all. So with that, I'm gonna go ahead and pass you guys back to Greg, who's gonna close us out and lead to our Q&A. Yeah, thanks, Chris. It looks pretty cool and straightforward that we can just use AI to help us do RL with AI. I mean, that's pretty cool. Now, if you didn't catch it, what we're really talking about here is we're actually doing this AI constitution, AI revision, and SFT to create the SLCAI data set, the SLCAI supervised learning for constitutional AI model, that at the end of the day is really helping us to generate this rewards data set. This is the crux of constitutional AI and of RLAIF. Now, beyond this, we simply have to then go train our reward model, set up our PPO scheme, and run through the policy optimization exactly the same way as we did with RLHF. So remember, sort of bring us back to the beginning here. The problem with RLHF, really the part that was the problem, was that it was kind of expensive. And the paper that came out last year, September, 2023 from Google, the RLAIF paper, and then was revised in December, just a few months ago, they came in and they said, you know that we can do this even, remember, if the model we're doing the assessments with is the same size as the policy model. And more potentially importantly, we actually don't even need to distill preferences into a separate reward model. We can simply directly prompt the LLM to provide rewards. And this can outperform this constitutional AI setup. That's pretty cool. Being direct. Now the constitutional AI setup, of course, the core of it, the point of stage one is to create a model that can then be used to create the harmlessness dataset. That dataset is the piece that's used here to actually generate the rewards model. So what we have finally in conclusion is we have, if we're doing RLAIF, aka constitutional AI, we're creating the revision dataset, we're fine tuning the SLCAI AI. We're creating the revision data set. We're fine-tuning the SLC AI model. We're creating the harmlessness data set. And the crux of the matter is we're doing all of this work to use AI to build the rewards data set, which we then fine-tune the rewards model with, and RL still comes in at the end. Now, Google extends this work about a year after Anthropic to direct prompting rather than using a fine tuning of a rewards model. Being more direct is going to be a theme for us as we continue our alignment series next week with, or in two weeks weeks with direct preference optimization. With that, that's a wrap for today. And if you do have any questions, please drop them in the Slido link. We're happy to have discussion with you guys, whoever's joining us live. What have you got on alignment? Let's go ahead and chat about it. Chris, all right. Time for a little Q&A. And I like this idea that the crux of all of this work that we're doing is just to create this data set that we could actually just go pull off the shelf from Anthropic. But the hard part about that data set was actually making it, right? That was sort of the whole thing. Yeah, it's so interesting that, like, it is the case that it's so much work to generate that data set. But the actual, like, the process of doing it, like, the code is not that difficult. of doing it like the code is not not that difficult it's just the you know the time and the the uh the spend that you have to to do to get like a very huge performant data set is uh makes the off the shelf shelf options pretty attractive for cases where they exist for sure right right right right yeah yeah because Because even like actually just running inference all those times to generate and then to perform the critique and then to generate another and then to do this over and over to have a big data set. All of this is actually cost money as well. Cost time too. Whereas you can just pull that one off the shelf today and maybe just use it with your model straight away. Right. That's exactly right yeah so we've got uh manny yes we did have a part one uh it was the rlhf and we did it a few weeks back we're going to sort of separate the alignment series each by two weeks so dpo is coming up next and then um question rlhf sounds amazing and almost too good to be true what are the limitations with our aif rlaif sounds amazing and almost too good to be true are there limitations and what's the what are the weaknesses of this particular approach beyond sort of it still costs money that we just kind of covered it does still cost money so it's not free right it's not free lunch or anything like that it costs much less money typically than you would pay human annotators. And the kind of quality of data that you get out the other side is probably winds up being a little bit better because of the volume of data you can get for the same cost. But a couple of limitations are it still does use this rl based approach where we're shifting the problem into the rl space uh in order to to solve it which does mean it comes with training stability issues now uh the rl hf event we did a couple weeks ago does talk about ppo as a method of training there are more stable ways that we could train uh and include this information but even still we're always kind of butting up against this training stability problem uh and training stability problems could result in larger costs right if you you know if you're trying to train a model and you the training fails a number of times right you still have to pay for all the all the failed runs so So, um, other than that, I would say it's weaknesses. This is not used useful for really macro changes. Like, uh, it's not going to get your model to be like really good at physics or math. It can be part of that process, but you're going to have to do a lot of heavy lifting to get to the point where RLAIF becomes, you know, useful enough to, to really like be amazing. So you can always use it and it's always going to improve things, but the closer you are to, to an aligned model before you start this process, the kind of bigger gains you'll get from RLAIF. So maybe those are some limitations. Yeah, yeah. So, you know, I wonder if you can just sort of like riff for a minute on like this. So this Google paper tries to take this to the next level. And basically they're like, hey, you don't need sort of this rewards model anymore. You can be more direct. Can you talk a little bit about how you interpret that and what the easiest way you think is to sort of think about this sort of first step in more directness as we head down the alignment path? Yeah, for sure. So like, okay, let's think about what we're doing, right? We're creating a model and then we're using that model to make decisions between two responses, right? We're saying a model and then we're using that model to make decisions between two responses, right? We're saying this response is preferred, this response is dispreferred. So that is kind of an extra step in a way, because if we instead had a model that just gave us like, say, a score between zero and 10 or something like that, right? Really any kind of score that communicates some range of not good, too good, right? Then we don't really need to train the reward model. And we're already trusting the model to capture our preference. You know, we are implicitly trusting it because we let it create a reward dataset, right? We let it create that preference dataset. So we're already saying this model is a useful standard for a set, right? We let it create that preference data set. So we're already saying this model is a useful standard for a human, right? So why not just cut out the kind of middleman preference model data set part and use this, you know, either constitutionally changed train model or in the RLAIF model, right? Just the off-the-shelf LLM. Why not just trust it to be the human standard and generate the rewardlaif model right just the off-the-shelf llm why not just trust it to be the human stand-in and generate the the reward scores directly right why would we why do we need to do this extra step now there is some utility we can fact check it we can having the data set means that we can kind of go through it comb through it remove responses we definitely don't agree with or we can pay human annotators much less to review a data set versus to create one. So there's a lot of different kind of, you know, levers to twiddle here. But the idea is, if we trust it to create that preference data set, right, why not just trust it to create the scores that we use during our alignment training, right? So PPO or the version that Google used in their paper. Yeah, yeah. So when we do that, then we need to be more specific in our prompting then. We want to give very, very specific instructions like generate one, then generate another, then assess them based on these couple things think through before you provide a response but but it's all sort of um self-contained in a single in a single generation versus many many many inference calls yeah yeah that's a great way to think about it and you know i'm kind of struck by you know when when uh open ai was going from gpt one to gpt two they were like well we can do unsupervised and then we can supervise fine tune and then they're like well actually if we just if we just directly sort of prompt the unsupervised pre-trained model without supervised fine tuning it works pretty well too this sort of zero shot task transfer idea that turned into in context learning and sort of what became prompt engineering by GPT-3. So there's sort of like, it almost feels like there's a little nested lesson in here that we're relearning at a new layer of abstraction, something like that, right? What is the field if not just constantly relearning the same lessons over and over again and applying them in new ways, right? Yeah, yeah yeah right right right yeah it's uh more meta always more meta okay friends so how many different models lms are involved in these various fine tuning processes helpful llm laura llm opening eye scoring llm how am i supposed to keep track of all these llms you know and sometimes it's just one of one lm but you're using it for 16 different things like uh yeah like what what what advice do you have for people to try to keep this stuff straight are there primary categories that we can put these things in boxes yeah i think so so so number one we want to kind of think of it in this way, right? We have our base model and then, uh, you know, we'll, we'll ignore, uh, constitutional learning for a second. We have our base model. Then we have some kind of, uh, reward model, right? Uh, and then we have our policy model. So that's, that's kind of like the three that you need, right? When we introduce constitutional AI, we also introduce the idea of this supervised constitutional AI model, right? So that is going to be, it's going to wind up taking the place of our policy model after we create it. But it, you know, until then it's there. and it's also going to be our feedback model so in reality a lot of these models can be the same which is even trickier I understand but you really only need the base model the reward model the policy model and if using constitutional AI will create a feedback model which is going to be our SLCA model. So there's four distinct models. The reward model is the most flexible because it can be anything that takes in text and outputs a score that's aligned with our preference, right? So it can be, it can even be an API, right? It doesn't, it could be whatever you want. It could be a, just a function that randomly select score basically if we don't do that, obviously, but like it could be. Whereas like those the base L11, the policy or are likely to be LMS using that kind of traditional decoder only network architecture. Okay, so I guess I guess I want to sort of end on this idea of more directness. So the next thing that we're going to cover is direct preference optimization. And, you know, here we saw that to go from sort of RLHF to RLAIF, we're like, well, how about we cut out this like sort of more complex piece, this human, like, man, dealing with humans, such pain, right? Then it was like to go from sort of constitutional AI to RL, AIF, it's like, how about we just like cut out a couple of the AIs, you know, just more direct. And so when we're talking about direct preference optimization, for people that have stuck around today, what's sort of a nugget of how to think about what's being cut out and what exactly we're being more direct about? Yeah. I think the easiest way for me to think about and communicate this is that we have this weird process that we need to map this one, whatever you're going to refer to it as, and then we have to allow it to interact with a different domain of machine learning. In this case, reinforcement learning. So the idea of DPO is we cut out that bit, right, where we're translating this problem to another domain, which leads to stability issues, which leads to a number of things, right? If we can keep it all in the place we're currently living, right? We don't have that translation error. It's wrong to say translation error, but it's a useful way to think about it. But yeah, I would say that's where we're cutting out. We're cutting out that translation from kind of our LLM, how we typically think about training LLMs to a totally different machine learning domain. Man, I just love how this field is maturing. We continue to see indeed the simplest solution is the best over and over and over again. It feels like the classic return to linear regression when you're in production or something from classic ML. Chris, man, it's been a blast hanging with you for this Q&A sesh. We'll see you back next time when we're back next week on YouTube live again. So thanks everybody for joining us today. If you haven't gone ahead, please go ahead and like and subscribe. Definitely, we want to keep bringing content to you that you value. If you like this session, dig into our community a little bit more. We've got a Discord. We're really, really on a mission to build the world's leading community for people who wanna build production LLM applications. We'd love to have you. So join us today. And if you wanna tinker on your own, and I know there was a question in the chat just a moment ago, looking for an old notebook, you can actually find all of our old notebooks in a nice table in our awesome aim index on our GitHub. So give that thing a star and follow along there if you want quick access to all of our notebooks, which you can always find in the first comment of our YouTube videos. And then finally, if you're looking to accelerate your LLM application development from prototyping to production, check out the AI Engineering Bootcamp, our industry-leading cohort-based live online course where you can fill all your skills gaps that you might have related to building, operating, and improving LLM applications in production. Lastly, if you have feedback for today, we'll drop a feedback form. Luma will also send you one. And that's a wrap. Until next time, keep building, shipping, and sharing, and we'll certainly do the same. We'll see you back next week, everybody. Have a good one.
Reinforcement Learning with AI Feedback (RLAIF) | Constitutional AI
GPT-4 Summary: Dive into the cutting-edge world of Large Language Models (LLMs) alignment with our latest YouTube series! Our second event zeroes in on Reinforcement Learning with AI Feedback (RLAIF) or "constitutional AI," an innovative method designed to overcome the high costs associated with human data collection in fine-tuning LLMs. Discover how RLAIF utilizes an AI-generated "constitution" to evaluate and refine responses to harmful prompts, paving the way for more ethical AI interactions. We'll walk you through the entire RLAIF process, from crafting an AI constitution, generating critique-based revisions, to the advanced training techniques like Supervised Learning for Constitutional AI and Proximal Policy Optimization. Plus, get hands-on with our Google Colab notebook, offering all the code you need. Don't miss out on this opportunity to explore the nuances of AI alignment, the interplay between RLAIF and RLHF, and the practical steps to harness AI for creating safer, more helpful digital assistants. Join us live for an insightful journey into the future of AI ethics and alignment!

Event page: https://lu.ma/llmsrlaif

Speakers: 

Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/gvvQ9NXEq6RZrWSj6