 A-whiz. So this REFT, this new reasoning with reinforced fine-tuning, is it basically just RLHF or RLAIF wrapped in a different package? It shares a lot of similarities with those, yes. Okay. And we've also got this sort of FT, this fine tuning thing, but I thought we were doing fine tuning in the other alignment methods we already looked at. We are, but we're doing more of it, right? Aha. So if I ask sort of what is the difference between this REFT and the previous methods we looked at, what would you say? Well, we spend less time in the SFT step. So we don't actually fully supervise fine tune the model. And we allow during that PPO training, we allow our model to explore multiple options. And then we use that and we consider that when we're updating our ways. Wow. Okay. It sounds like we got a lot to untangle today. You ready to get into it? You know it. All right. We'll be back in a little bit with some discussion before we get into the demo. Thanks Wiz. Today we're going to try to tackle this new idea of reasoning with reinforced fine-tuning. It's kind of a remix of ideas we've covered before. And if you're familiar with alignment techniques like RLHF and DPO, it's going to be one you can get a handle on pretty quickly. Today, you'll walk away understanding exactly what we're doing to combine chain of thought with RL to kind of produce this interesting new higher performance way of doing math problems. And of course, we'll walk you through the concepts and code you need as usual. I'm Dr. Gregg, that's the whiz, and we'll go ahead and get into it for today. If you've got questions, please drop them either in the YouTube live chat or in the Slido link in the description box, and that we'll share right now. So let's go ahead and rock this out. Alignment with R-E-F-T. And of course, this is the continuation of our alignment series. And each time we meet, we're always aligning our aim at AI Makerspace. So by the end of this session, you will make sure to understand just how REFT can enhance reasoning capabilities using just a single set of training questions. So you can imagine that this has potential impact beyond the math problems we'll look at today. We'll get a handle on how REFT includes two primary phases, not fundamentally distinct from other alignment techniques we've looked at at the warm up and the RL training phases, and then we'll show you exactly what the code looked like that was released in the paper and how we can leverage it to do our own REFT. All right. So first we're going to kind of contextualize this with the previous alignment series, RLHF, RLAIF, DPO stuff that we've done. We'll get into REFT. We will talk about the two phases. And then, of course, we'll do the thing. And we'll answer the question by the end of the day. Does it do the thing that we really needed it to do? Spoiler alert, the answer is yeah. Yeah, it appears to. Appears to do exactly what it says on the tin. So let's contextualize the discussion. If this is your first time joining us, if you haven't seen the rest of our alignment series, you might want to check that out after we give you some trailheads today. We've covered alignment with reinforcement learning with human feedback, reinforcement learning with AI feedback, and the emerging new best practice standard, direct preference optimization. You can find all of those at this link on our YouTube channel playlist. The purpose of alignment, though, in general, that we've seen, and this is a plot from the Lama 2 paper, is to more or less reward good behavior. Good behavior. And that can be defined, and we have defined it so far, as behavior associated with being more harmless. We take helpful models and we make them more harmless. We've aligned models like in the Lama 2 paper using supervised fine tuning, SFT, and reinforcement learning or direct preference optimization approaches to kind of achieve this distribution of reward scores given a series of prompts that increases in its likelihood of producing a highly rewarded output. So this alignment, this rewarding of good behavior, it turns out you can actually align towards many different objectives. It doesn't simply have to be about being less toxic or less harmful. And in today's lesson, we'll see that this sort of alignment technique of REFT kind of straddles the line between alignment and fine-tuning. And what we're really doing is we're aligning the model using the same proximal policy optimization scheme we've seen before towards becoming better at solving math problems. How are we doing this? Well, let's break it down. The paper came out January, 2024, just a couple of months ago. And the big idea, this was from ByteDance Research. And the big idea is we're combining reasoning, in other words, chain of thought, reasoning, with reinforced fine tuning. So let's tackle the first piece here first. When we talk about chain of thought reasoning, we're referring to the paper that came out in January of 2022. This was from researchers at Google. And the idea is that when you answer a question, when you are, let's say, a middle school math student, if 1 to 500, and you get A, B, C, D, E choices, a regular kind of potentially student not trying that hard might just say, the answer is B. And then you'd be like, what? Did you just cheat how did you know that right but a student that's ready to sort of tell you exactly how they got to the answer well there are nine one-digit numbers from one to nine there are ninety two digit numbers from ten to ninety nine and so on If we add all these up, that's 1392. Therefore, the answer is B. You're like, oh, okay. You really do understand. We want our LLMs to be able to do this too. And although this chain of thought paper went far beyond math problems, we're going to focus today's lesson around math problems. And since this was work done by researchers at Google, what they did is they, in terms of this math problem, data set called the GSM 8k, which we'll look at more closely in just a moment, they took the fine tuned GPT-3, the Big Daddy 175 billion parameter model. They also took their GPT competitor, Palm, their Big Daddy model, the 540B, this was in 2022. And they they tried to sort of pin it head to head and in head to head prompting not so much. But if they included this chain of thought, this thinking through on the math word problems, they could get a much, much better score. And of course, this is how you know that young learners are understanding the math. This is how we all came up in the game. Turns out this grade school 8K math data set is originally curated by OpenAI. And you can find it at this GitHub link here. Problems like Mrs. Lim milks her cows twice a day. Yesterday morning, she got 68 gallons. And in the evening, she got 82. The next morning, she had 18 gallons fewer. After selling some milk in the afternoon, she only has 24 gallons left. If she's selling each gallon for 350, how much money did she make? And you can kind of track, okay, well, 68 plus 82, 68 minus 18, that's 50. You add them all up. You get this subtraction of the remainder, and then you get the final dollar amount she made. We want our LMs to be able to do this. And since the paper came out in 2022, we sort of see that the GSM 8K dataset, this is from papers with code, you see the Palm 540B back here in 22 did pretty well. You see that GPT-4 has kind of been crushing it. That's really not the point of today, but it's worth noting that this is at about 97% accuracy today. Models can do, the best models out there can do very well on these types of problems. And in fact, this is at the middle school level, grade school math 8K, 8,000 questions, grade school math level. Okay. So this is the idea of the chain of thought piece. How do we combine it then with this reinforced fine tuning piece? That's really the crux of the matter today. Well, here is the figure from the paper. In stage one of reinforced fine tuning, what we're doing is we're doing supervised fine tuning. Now check this out. Question X, Wang earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. Okay. Well, she can only earn 10 bucks then, right? right? The chain of thought piece of this E noted here is the piece that we're doing fine tuning with in addition to the question and answer, all right? So the fine tuning that we're doing across if you supervise fine tuning epochs is input, output, but also including this chain of thought. So we see this across a number of epochs. Simple, okay? All we're doing is we're doing some SFT. Classic. Adding a little chain of thought action to it. What we do in the next stage is the next stage is this REFT stage. So once we're nice and warmed up, then what we do is we go and perform RL on the same training data. And this is the sort of image they provide in the paper. A great figure, in my opinion. And we see this sort of question, this on policy sampling, this X, E prime, Y prime terminology here. And then we see this golden reward. Y equal, equal Y prime question mark. This is potentially a bit nebulous as to what this means straight away from the figure. And so we're going to try to break it down in the context of what we've already talked about in previous events. So that's super clear exactly what's going on. talked about in previous events so that it's super clear exactly what's going on. The idea is that this REFT starts from the warm-up position that was done with SFT and performs this reinforcement learning using a proximal policy optimization scheme. So we can ask the question, and we'll ask this a few times today, is it really that different than RLHF? Well, let us recall RLHF for a second. Step one was simply instruction tuning. We wanted to get more helpful. This is from the Instruct GPT paper. We call this training a supervised policy in RLHF. And of course, we're just doing supervised fine tuning. Step one, supervised fine tuning. Step two, train a rewards model. This was something that allowed us to decide which response came out less harmful. And so we would go through this training of a reward model piece. And finally, the policy optimization scheme where we're updating the weights relative to the reference model. You could view it like this, but as we've started to become accustomed to, if you've been following our content, we have been viewing it a little bit more like this. Prompts go in. We run them through an initial model, a reference model, and a tuned or policy model. This is the one we're aligning. You could call it our aligned model. And then we're checking if these models are close enough, not too far apart. This is that KL divergence that we're looking at. Then of course, we're leveraging our reward model that we've trained to get a reward score from the tuned policy or aligned model prompt before we update the LoRa weights in our tension layer and go through this process even more times for a given number of iterations. For more on RLHF, check out our RLHF event. We've gone into a lot of detail there. We can also recall our AIF. And we saw something similar there. We saw that in stage one, look at this, supervised fine tuning. We called this the SLCAI model because we were doing constitutional AI in this particular event. And stage two was the reinforcement learning piece. Look at that, SFT, then RL. And again, we're doing the constitutional AI. This is the figure from that paper that we go into a lot of detail on in that particular event. And of course, there was this sort of response critique revision piece that is part and parcel of constitutional AI, but a bit unnecessary for today's discussion. constitutional AI, but a bit unnecessary for today's discussion. But perhaps it's not that useful to look at everybody else's reinforcement learning, system diagrams. So let's go back to the one that we prefer. Proximal Policy Optimization Scheme for RLAIF looks something like exactly what we just saw for RLHF, only the reward model was built and trained leveraging AI. Now it's worth noting here that further RLAIF work done by Google that built on the work done by Anthropic showed that you didn't have to actually even train a separate reward model. And that was pretty interesting. And we went into some depth there in our RLAIF event. Check that out for more info. Finally, we can recall DPO. And DPO made everything even more direct, even more direct than not even training a reward model, but instead prompting an LLM to use it. We weren't even using an LLM anymore. Rather, we're using an analytical mapping. We're using a math equation to establish our rewards. That math equation looked like this loss function. We won't go through the details, but of course we have the details available in that event. Again, let's put this into context of the diagrams we're used to and the direct preference optimization scheme looks something like this. Now you sort of notice we changed the name here from initial and tuned or policy to aligned. We're talking about the same exact thing. Prompts go in. There's a reference model. There's an aligned model. We're checking KL divergence. These are not too far apart. Then we're calculating a reward score for both. We're plugging that into a loss function. We're updating lower weights. a reward score for both. We're plugging that into a loss function. We're updating lower weights. And of course, for more on DPO, you can check out our DPO event. So in light of all of this, let's reconsider REFT. We have stage one, SFT. We have stage two, reinforcement learning. Sounding pretty familiar at this point. So what we might say here is we might say, well, we're kind of doing exactly the same thing for our EFT as we were doing for RLHF and for RLAIF. We are simply putting in prompts, we've got a reference model, we've got an aligned model, we're checking that they're not too far apart, KL divergence again, it's still PPO, and we're leveraging this reward function. Also it's worth noting that in the paper implementation, they were not using a LoRa, a PEFT LoRa implementation of updates, which we'll get into in the demo here, but rather they were updating the full weights. This provides some challenges that we have to overcome in the demo that you'll see shortly. But for right now, the key is this reward function. What is this reward function in Reft? Well, the reward function is very simple and it's pretty cool that it works. For a given prompt, we're gonna generate multiple responses. And we ask ourselves, is the correct numerical answer contained? is the correct numerical answer contained? So we're gonna generate a full chain of thought response and we're just looking for is the right answer contained anywhere within that chain of thought response? If yes, reward that. That's good behavior. Good job, LLM. If no, you get no reward. Now here's an interesting little tweak as well. If there's at least one numeric character present, we give it a partial reward. So if it decides to generate at least one number, it's kind of a reward. So just to triple down on this idea, just as in some RLAIF methods, not constitutional AI. And in DPO, there is no separate reward model. There's just a reward function similar to the loss function function directly used in DPO or to the prompted LLM that you could use in some of the more streamlined RLAIF schemes. So given this context, I want to invite Wiz back up to the stage. I want to have a quick discussion here. So there's like, seems like everything's the same, but not quite the same. So the first question was, is, is this warmup thing happening? Is this just SFP? Is that, is that all that's happening? Is there something else going on under the hood here? Nope, nothing else going on under the hood. It's just SFT. I mean, the, the thing that we might think is different about it is that we don't care to train it fully, right, to completion. We're happy to just do a couple epochs. And that's why we're going to call it warm up, right? This idea that we're not going to, we don't, we don't actually care if that SFT step results in a model that performs, you know, if that SFT step results in a model that performs, you know, as, as well as it could on our task, we don't care to wait for convergence. We, all we care about is that we kind of get it heading that direction with SFT taking those big steps so that we can be ready to take those smaller steps through the, the PPO training process. Okay. Okay. So generally when we're doing SFT, training process okay okay so generally when we're doing sft we're looking for a convergence in the loss function is that right we'd like to see that yeah yeah yeah yeah that's the idea yeah that's the idea okay so here we're not concerned about that at all we're just doing some fixed number of epochs just to prime the pump kind of thing on this? That's exactly right. Yeah. Okay. Okay. All right. That's pretty interesting. And then next question. So we saw that it is the same PPO scheme. So is this basically the same exact thing? Is REFT just RLHF? Kind of, I mean, yeah, there's some, there's some, I would say, like, small differences, right? Which is that we, with the traditional RLHF, we're just going to kind of, you know, compare the distribution that our policy creates versus the distribution that our reference makes, and then make some adjustment to our weights. Whereas in our EFT, we kind of do this guided process where we actually verify if the model is generating correct responses and not just, you know, a correct response, but if it can kind of express that in different ways, if we can get kind of these combinations of correct responses which is going to lead to better generalizability on that specific task so whereas rlhf is going to laser focus us in to be very very performant at this task but you know there's maybe a generalizability trade-off uh you're going to pay for that. And REFT is trying to solve that specific issue. Ah, okay. So when we're using RLHF as we've used it in the past, I mean, we're not making generalizability trade-offs, right? We're just aligning it in general, right? To sort of anything you might use it for, or we're aligning it after we've made those generalizability trade-offs, right? It's sort of like, we're not, well, fine-tuning it. Although we are. I would say it's actually flipped. So the idea is that with RLHF, right, we're going to get very good at this one specific task, and we should see that it's very good at that specific task in those specific settings, but we lose generalizability because when we're training, we want it to be very good at that one specific task versus REFT, where we allow it to see a sample of potential outputs and to to score and adapt to those uh you know distinctly so we can have kind of uh better we're we're more likely to have better generalization from the reft method than we are from rlhf oh wow okay interesting interesting so could we use both methods to So could we use both methods to align towards other things than harmfulness? Both are yes. Okay. Not combined, perhaps, but definitely independently, you can use both to go for helpfulness, harmfulness, or you can get just better at making JSON, right? Like it's the domains that you can apply these techniques to are, you know, they're quite a bit. Okay. So to continue to try to disentangle this web of alignment, should we expect like everything else in the world that this will be DPO-ified this reft thing in the near future and would that even make sense to do for this particular method it can make sense in the offline version uh i don't know that the online version is going to make a ton of sense so i'm sure you can you can you know wrangle the math to get there uh i don't see the clearest path forward but i'm sure someone clever will be able to come up with that though i don't know if that's going to be something that happens soon or uh you know quickly uh definitely though the offline version because it is so much i mean is exactly rlhf we should be able to port it to dPO with little issue. Yeah, I mean, the reward function versus loss function thing, it seems like it's amenable. You know? OK, OK. All right, cool. I hope that was useful to everybody out there. It's certainly useful to me to keep plugging away at some of these questions as I try to get the big picture. Thanks, Wiz. We'll see you back in just a little bit to do the big picture. Thanks, Wiz. We'll see you back in just a little bit to do the full demo. So in the paper, we were told in terms of today's build that all the code was provided. This is from the abstract of the paper. And they mentioned that the code is publicly available. And they did this not just on the grade school math 8K data set, but also on Math QA and an additional math data set. And they saw consistent performance improvements. What we're going to do today and what we need to do, because we don't have that PethlORA approach built into the code that was publicly available. We're going to pick up a model that's actually an SLM off the shelf. Just a little baby 125 million parameter. This is called Galactica from Facebook. And this was trained on a large scale scientific corpus. And again, the reason we're picking this up instead of a Mistral 7b is because without PEFT-qLORA, without the quantization, without the parameter efficiency, it would just be crazy to go through this entire process with a full fine-tuning of a large language model. So we're going to use this model, Galactica 125 million. We're also going to use the data set of the Math QA. This is a large scale data set of math word problems. It's not anything very different than other math word problems you'll find out there. Although it's not going to use multiple choice answers during our SFT or REFT parts of the process. Let's just recall that what we're going to do is we're going to iterate several epochs on the training data, warm it up. Then we're going to start with that warmed up model. We're going to perform reinforcement learning via PPO using not a separate reward model, but rather that reward function that helps us decide, should we reward it? Is it right? Should we not reward it? Is it completely wrong or should we partially reward it? Does it include a Should we not reward it? Is it completely wrong or should we partially reward it? Does it include a number? And with that, I'm going to pass it over to my man Wiz to show us how to perform R E F T. Over to you, man. Yes. Thank you, Greg. Okay. So the actual code is pretty straightforward. I mean, it's given to us, so that makes it always makes it easier. But it works and that's great. We have to do a few fiddly things in order to make it work in a cool environment. And I do want to be clear that I did this on a 100 instance and we did wind up using 31 gigabytes of GPU RAM during the PPO step. So while you can fit the SFT step on about 16 gigs with the setting or hyperparameters we're using, the PPO step is actually going to be quite a bit more memory intensive. You could reduce kind of, you know, some hyperparameters even more in order to squeeze that out, but it will take proportionally longer depending on which chip you're using. So just to get that out of the way straight away, I did use A100 instance, which requires Colab Pro. After that, it's pretty straightforward. We just got to get the code from the repo. So the repo is here. Basically, you know, it shows us exactly how we would run these. And it works. It does the thing. Doesn't have specific instructions for the small model, but it's the same. We just change it to the small script versus the regular script. We're going to cd into that directory so that we can run all our scripts with the right paths. And then we're going to install some requirements. We're going to install some additional requirements that don't get picked up properly. So we'll need deep speed. This is the distributed framework they're using. The examples are meant to be run on multiple GPUs. We're changing some things so that they don't do that. But if you wanted to run this, say, on a local cluster that you had or at whatever hardware you have available to you, it is using DeepSpeed, so it'll adapt to that well. We're going to also pick up a few additional dependencies that don't get picked up properly by the requirements, and then we do need the specific version of TensorFlow GPU. And so we get it. We're gonna use wandb in order to do our experiment tracking, so we'll get the wandb API key so that we can look at our, how our training went. And then we're gonna use HuggingFace. So we gonna also do our notebook login, which is going to let us be in Hugging Face, make sure that we have access to the right repositories, and we can push anything that we want to push. So there's two steps, two steps really, right? There's this SFT warmup step, and then there's a PPO reinforcement learning step. The first of those is pretty good. So the SFT warmup is exactly what you'd expect. It is, you know, TRL, behind the scenes. So the library that we've used to do this before. And all we have to do is make some modifications to the script file in order to run it in a Colab instance. What we're going to do is we're going to change some of these hyper parameters so that they better fit our hardware we're going to go ahead and do just one epoch the paper suggested about four epochs so and you're gonna likely see better results if you do that so keep that in mind we just use one epoch to keep it light today if you do that. So keep that in mind. We just use one epoch to keep it light today, but you'd want to use three to four if you're following the paper exactly. We do have to change the model path. There's been some updates since the code was released. So we're using Facebook's Galactica 125. That's this model Greg was talking about. Basically just a 125 million parameter language model. That's pretty good at the task it's meant for, which is kind of these science-related tasks and language generation. We're going to also look at basically only doing one process. So we're not going to be using the default of eight since we're just working with our Colab instance. And everything else we can kind of leave the same. So the way you would do this, to be clear, is you would head into the directory that holds your cloned repo. We're going to go to our examples directory, our small models examples, and then we're gonna go to, we're gonna be using a Math QA today. So we'll double click on the script. And then we just wanna make sure that this file lines up with what we have here. You can just straight copy and paste this into here and it'll work great. So once we have that set, we're gonna do one more change and that's to kind of, you know, make sure DeepSpeed is aware that we only want to use one process. And so we're going to come into the DeepSpeed YAML and we're going to configure it so it only uses one process. Again, this is just due to the fact we're running it in Colab. If you're not in Colab, you don't have to worry about these things. Just set this to the number of processes that makes sense based on the number of GPUs you have in your hardware setup. After that, we just call the training script. And thanks to wand B, right? We have the ability to kind of see how training went. So we have everything from our kind of loss curve, which is, you know, looks pretty nice to our, you know, this is our learning rate as it progresses over the training. And then we also have our value accuracy. So we see this about 14%, right? So this is a, you know, this is meant to be done in four epochs, so we should see this trend up over time. Obviously with one epoch, we only have a single point. So this is the way that we're going to see it. But this is the advantage to using 1B. And you can see it kind of does what it's supposed to, right? Trains the model. We love to see it train the model. The next stage is to do the PPO step. So this is the proximal policy update or authorization step. It's the cool part, right? We have to make a few changes. So number one, because we only did one epoch, we have to change the actual checkpoint that we're pointing to. So this is, if you follow this notebook exactly, this checkpoint will be correct. But if you've made some adjustments or you say you've done it for four epochs, you'll have to make sure that you highlight the correct directory. Now that can be found through your repo. And we're going to go to a basically a directory here. And that directory is going to have our outputs and we have our math QA. And then we'll select the directory associated with the training checkpoint that we want. You can see here that in the notebook we're using epoch one, but if you did four epochs, right, you can assume that we'll be using a checkpoint that's associated with the fourth epoch. Or if you had some kind of crazy, you know, loss explosion, you might want to select a different checkpoint depending on how training went. Once we have that, again, the only thing we're changing is kind of this num processes. And we're also changing our batch sizes. So I found that with 8, we used that 32, 31.X, sorry, gigabytes of GPU RAM. So if you're on a t4 instance right you might want to uh even even have this uh so that you can fit this within uh you know the 16 gigs but for the a100 we can use eight safely not run into any out of memory issues which is what we kind of want right um you'll also see that we have kind of our normal stuff, right? So we have our KL coefficient. This is that idea that Greg was talking about where we're clipping our models together so we don't have too much drift from the base model. We don't want to lose the ability to do our task. We just want to get better at it, right? As well, we have this VF coefficient. This is useful in this specific loss that the REFT process uses. So because we're using a custom loss function, we have this coefficient that we can kind of manipulate to determine how strongly we're training the model. And other than that, you're good to go. We just let it rip. And you can see that we get kind of, you know, this final result. And we can see that these numbers are okay right off the hop we notice the accuracy is better i mean that is ideal right uh even though this is very much a kind of toy example where we're just doing one epoch uh and then we're doing one epoch of one ppo epoch in the in theL step, we still do see that that, you know, that number goes up. However, when we look at the trading, we see a slightly less clean story than we did with our, you know, obviously with our original trading. So you can see that while we did have kind of this okay-ish loss at the start, we do wind up just kind of blowing up and bouncing all around um this is to be expected right we we we only did one uh epoch of the sft warm-up and then we only did one uh epoch of one ppo epoch for the uh the actual rl step so this is you know kind of what what is to be expected. We do see that the model, it does learn at the start for the VF loss, but then we kind of blow up. So keep in mind that this is, you know, to be expected, given the small train that we did. But still, we do see that despite that, we kind of have a better, you know, accuracy on the evaluation set, right? So 10% total increase. So a not bad, right? So let's look a little bit into the code here and see kind of really what's happening behind the scenes. So there's a couple things I want to highlight from the code. So this is obviously, you know, there's quite a lot of the code, just to be clear. So we're not going to focus on it too much. But you can see that this idea of correctness comes into play, right? So this is what Greg was talking about. We were talking about how we determine our reward, right? Instead of having a reward model, we're just using this reward function, which is going to give us kind of points based on if the answer is correct, if it is a little bit correct, or if it is not correct. The idea is if we have this correct value, right, so this is basically all this conditional is doing is saying, hey, is our extracted answer the same as our target value? And if it is, we're going to reward that bad boy with a one, right? And then is, if it's not, but we have an extracted answer, we're going to reward it a little bit, right? So in the case of our model, what we're going to think about that as, right, if we generate the correct number, we're going to get one point, right? And this is to be used in our PPO steps. This is the reward we're talking about, right? So we're going to get one point if our answer is correct. And then we're going to get 0.1 points if we have a value, but it's not correct, right? So it's like, okay's okay, getting values is where we want to move to. And then if we don't have a value, we're not going to reward it at all. And I want to show you guys something that becomes apparent in the actual training process. Okay, so at the beginning of our training process, if we scroll all the way to the kind of the top here, you're going to notice that there's a lot of errors and a lot of issues with extracting an actual value, right? So you're going to see that kind of for the first like, you know, 100-ish steps, we actually have a difficult time extracting an answer at all. And that remains the case kind of throughout, right? Even into the 150th step, we're potentially running into issues where we're not able to actually get a value, right? But as we train, you'll see that this is going to occur less and less and less. And the idea here is that we are still rewarding the model for getting a value that works. And so our model becomes better over time at providing us with values. And so we get less errors. And as you can see, as we get towards the end, we do very well at this task. And so, you know, that's the kind of power of this process. Even though we're not necessarily getting super close to the correct answers, we are still getting better at giving answers, which makes it easier for us to get to the correct answers. And that's thanks to, in part, what we were just looking at here, this idea of the reward function. We're also going to look at one more piece, which is this idea that we are using that custom loss. And so when we look at our actual, you know, where we're going to be able to compute the loss, we'll see that there is a, it's a custom implementation. Basically, all we care about is that we, you know, Basically, all we care about is that we, you know, we're going to have this idea of thinking about our loss a little bit differently, right? So instead of having that kind of, you know, specific PPO loss, we want to take into account the fact that we've explored multiple paths. We have multiple different reward signals. multiple paths. We have multiple different reward signals. And so we want to compile those and allow our training to benefit from that, you know, diverse idea of correctness. And that diverse idea of correctness is exactly what lets us, you know, get more of that generalizability that we aren't getting from RLHF. So instead of just considering if this is correct or incorrect or whatever it happens to be, we have the ability to say, like, this is a correct flavor answer, but this is also a correct flavor answer, right? Even though the actual generations might be distinct, and that lets us adapt better to these tasks. And so that's kind of one of the ideas of this REFT process and what it's meant to do. And so that's it. You can see here, we just kind of combine these losses and we're gonna scale the second term of our loss by that coefficient that we set in the notebook. So this is that coefficient here. And so this is the idea, right? That's all we have to do. It's basically RLHF with extra steps, but they're good extra steps and they really help us to understand and they really help us to get our model close to where we want it to be while still being able to adapt to all of the potential tasks that could be in that space. So we're not just getting much better at one specific task, we're getting better at a general task set, and in this case, math. And we can see see even though our numbers are kind of wild and wacky, we do get to that 23.03 versus the 13 on the base model. So good job, REFT. With that, I'll pass you guys back to Greg, who will close us out and take us to some Q&A. Yeah, awesome. Thanks, Chris. Awesome to see that wand B flexing on the good old-fashioned tracking the weights and biases, the parameters actually in the model. So a little old-school throwback there to bust that one out. Really, really enjoyed that. And, you know, at the end of the day, we learned that phase one and phase two, this SFT RL thing, this is the sort of pattern that we can match across the RLHF, the RLAIF, the REFT, and it's the exact same PPO scheme. We're also taking this idea from the DPO scheme of not leveraging a reward model, but instead leveraging a simple function. And we're using that in the REFT process. The reward function, as Chris just walked us through, has very simple rules. Correct, we get that one. If we're on the path, a little bit correct, we get that 0.1. And as we come to the end of our alignment series, the future of alignment is very bright. As we've talked about, there are many ways to align towards different tasks. And we're excited to see what future techniques, what future things people will be aligning to are going to come out next in 2024. So if you have questions, I want to go ahead and throw the QR code up on the screen here. Please do upvote them and add them to Slido. Or you can, if you'd like, go ahead and throw them into the YouTube live chat. So I want to bring Wiz back up here, and I want to start off the discussion today by doing a little comparison. And this was work done at OpenAI that they released May 31st, 2023. And they called this process supervision. And this is where they went and they quote, rewarded each correct step of reasoning, not just rewarded this final answer. And this sort of was their idea on how to directly train the model to produce an entire chain of thought that was correct, rather than just an answer that was correct. And the title of this blog was Improving Mathematical Reasoning with Process Supervision. It seems like everybody's been paying attention to math problems. Everybody's been trying to improve LLM's ability to do math. How would you compare this process supervision technique to the REFT one that we just used exactly? How should we sort of situate that? Is this also sort of an alignment technique happening here? Yeah, definitely. I mean, it's part of it's kind of part of that ecosystem, right? I wouldn't say that they're exactly the same or they're trying to achieve exactly the same goals, but they definitely do have a lot of overlap. And there's an argument to be made in either case for which is the kind of of correct you know uh way to do this i mean as you said kind of math math problems right uh doing more and more complex math is is a goal that we have in our uh you know that we're working towards in a number of different ways. I think where REFT comes in is this idea that we're making some key assumptions that I think pan out very well, which is that we, you know, we can't, if we're getting the right answer, then the chain of thought steps we're taking are likely correct, right? Or are in some way more likely to be correct than if we don't get the right answer. And so I think that assumption is very powerful and likely true because it's unlikely the model will get to the correct answer while meandering off a weird path, right? Though certainly we could overfit it to do that. Whereas this process by open AI is more about making sure that we have the ability to reason through problems. And so when it comes to that generalizability trade-off, this approach is making very little concessions when it comes to generalizability, because we're actually learning how to think through problems as opposed to learning how to get to the correct answer. And so I do think that this, what we have on the screen now is ultimately more flexible, but that might make it a little bit less powerful, especially on the benchmarks that we're looking at today. Yeah. Yeah. Okay. All right. And I noticed, so yeah, that was just sort of a thought in my mind that we'd seen this before. Everybody's focused on math and it's not exactly clear how to do this right. I do like the sort of generalizability comment on thinking about process versus outcome supervision here and definitely an interesting space to continue watching. I wanna move to sort of double click in on this sort of rewarding of the partials that we're seeing in REFT. So I know Forenz asked in the chat and we have a question in Slido on sort of like this idea of exact match. And when we're talking about this on a token by token basis, it's like, we're looking for a number? Is that what we're doing? And what if it's a big number? How many tokens is that, that we're actually looking for in a sequence or it's it's just like if it's completely the right answer in the exact sequence you get a one if you have any number whatsoever in any token string you get a 0.1 is that really all it comes down to and is there a way to sort of expand on this with this idea of semantic similarity that we've been talking about in other events versus exact match? So, yeah, I mean, basically, so it is, we have to be careful of the language here because we don't, behind the scenes in the reward function, we don't care what tokens at all. We convert it to a Python float, and then we see if that float is the same as another float. So the actual token doesn't matter there. We do use our log probes to update, so it will be backported to tokens. But the idea is when we're doing this exact match is literally just seeing if the two floats are the same, right? The idea of having a different, you know, number being marked as partially correct is as I was going through the demo, right, showing that at the beginning, we barely even get comprehensible answers, right? We don't get values even. And so we want to reward the model because we want to continue to nudge it to like, hey, by the way, you should be producing a number at the end of this. Like, I should be able to get a value from you. And if I don't get a value from you, you're doing it wrong. And so we want to nudge it in that direction, right? So that over time, we're always producing our expected outcome, which in this case is values. But we don't want to reward it significantly, because we don't want it to be happy that it's just generating values. And so this is the idea that the correct value is, you know, significantly more rewarded than the incorrect value, even though we still want to nudge it that direction. The nudging. So the nudging sort of builds on this warmup idea that we had. It's sort of like we're, we're warmed up, we're moving down the path and we have to keep moving down the path. It's just one or zero doesn't allow us to do that. But the little nudge-a-rooney, the point one, okay, you're doing something right. You know, you're crawling your way to an answer. If you ask a kid, right, what's two plus two and they say five, you go close, but wrong. Yeah. You know, and it's the same thing for the LLM. Whereas if you say what's two plus two and someone says like apple, definitely wrong. Exactly. and on the idea of semantic closeness um that could be very useful for other domains absolutely right uh if if what we care about is semantic relatedness then we're kind of moving back into a traditional reward model system less so in this reward function space. The issue with math, right, is that the incorrect response will be incorrect every time. And there's not really a benefit, right, to rewarding, okay, well, you're close. Well, we don't care that you're close, right? In math, it doesn't matter that you're close to the right answer. It's either correct or incorrect. In math, it doesn't matter that you're close to the right answer. It's either correct or incorrect. And so I think we're less likely to see like a plus minus 5% close to the number consideration. But for other tasks, absolutely. Or perhaps in approximations, absolutely useful as well. Yeah. And I mean, even closeness when you get into the process supervision versus outcome supervision, the closeness is like which part was close. That's right. Exactly, right? And it's very, very unclear how you would define even being close on any given large problem set that span many domains okay so uh here we are wrapping up our alignment series so we went through rlhf we went through rlaf went to dpo now reft um what are your top level takeaways and what do you see on the horizon for alignment? And is there anything that you're paying attention to that the rest of us should know about? DPO and all of its variants are so cool. They're so good. They're more stable. They're easier to implement. Blah, blah, blah, blah. We go on forever. And getting in the last function allows us to tinker in a way that people are very used to in ML in these domains, right? So RL folks are very used to dealing with their processes, but we're very used to the loss function. And alignment is going to become table stakes at some point right uh it makes the model better and it's relatively low cost uh i think we're gonna see more and more of these techniques used and we'll also start to get kind of more evolved data sets that help us kind of get it to a specific point that it should be uh right so it's like here's your harmfulness helpfulness uh you know toxicity bias whatever x math math yeah json right like yeah yeah yeah right specific data sets you need we're gonna see more of those proliferate as well and uh at the end of the day you know in in a year i imagine this is just a part of the process. You know, it's just how we do data preparation and we do fine tuning. Then we do evaluation. We're going to sneak in a little alignment step there because we know we can get the model close. We really want it to be exact, right? And that's what alignment helps us to do. Okay. Okay. Yeah. Love that insight. So we don't have any more alignment methods. We really want to cover on the docket, but we'll keep our eyes peeled for things beyond DPO that we think are important for you guys to pay attention to. Just one quick final question, tactical question that came in last minute. Can the reward be negative in this method, in this ref method? I don't think we want it to be. No. There's, there's a, it doesn't seem useful in this specific case to have a negative reward. I I'm sure that there are processes that you would, would want to negative reward, but in this case, the zero is zero is fine yeah yeah like like how how bad is it to get the wrong answer on a math problem right something like that yeah it also does weird things with the the rest of the calculation so we just want to leave it as some non-negative uh value between i i don't think it has to be between zero and one but zero one's very very easy to work with. Love it. Okay. All right, Wiz. Well, that's a wrap. Thanks so much for your insights today. And it's time to go ahead and wrap up everybody. So thank you so much for joining us for another Wednesday live sesh. If you enjoyed it, go ahead, like, subscribe, and ring that bell for notifications about what's going on and the latest and greatest. You can find us every Wednesday on YouTube at the same time, same place. If you like this session and you're not yet part of our community, you should join today and start building, shipping, and sharing with folks aligned along the same path as you. And we try to make it easy to do that. In fact, we recently started open sourcing our six-month-old courses. This is something that we look forward to continue rolling out. If you haven't checked out our LLM Ops, LLMs in Production Cohort 1 course on GitHub. You can start learning there directly to get up to speed on your LLM application development from prototyping to production. If you're ready to like super accelerate beyond what you can do open source from our resources, we've got more stuff coming up in Q2 that is gonna be open sourced as well. Then please do check out our AI engineering bootcamp. We've got a couple of cohorts coming up. You get the opportunity to work directly with the whiz, myself, AI Makerspace peer supporters, and a group of engineers and engineering leaders that really are pretty legendary. Every single cohort. Certifications open up opportunities with hiring partners and consulting work throughout our community. And then finally, if you have feedback on today's event, please let us know. We'll drop a feedback form in the chat. You'll also get a ping from Luma. And until next time, keep building, shipping, and sharing, and we'll do the same. Thanks so much, everybody. Have a great week. See you soon.
Aligning LLMs: ReFT
In this event, we’ll break down the steps of ReFT, which consists of two stages: the warm-up stage and the reinforcement learning state. We’ll also discuss how the authors were able to achieve significantly increased performance on classic benchmarks like Grade School Math 8k (GSM8K), MathQA, and Simple Variations on Arithmetic Math word Problems (SVAMP) datasets.

​As always, we’ll demonstrate how to code the core aspects of ReFT yourself in a Google Colab notebook environment, and all code will be provided directly to attendees!

Event page: https://lu.ma/llmsreft

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/1mMvrSnsuDv9iEvRirykwx

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/QqHnx1EovmPUfcnc8