 All right, Wiz. So loss functions. Are these things in GPTs? Yes. Are they in embedding models? Yes. Do we need to use them for fine tuning? Yes. How about reward modeling like RLHF? Also there, yeah. DPO? Has loss. End-to-end RAG like the domain adapted systems from our friends at RC? What did you believe at loss? Vision models? Loss. Multimodal models? models multimodal loss does everything have loss for training in machine learning yes huh seems pretty important no yeah it's pretty we don't want to lose out on it. Well, everything is not lost. Today, we are going to drive down to the heart of this thing, see if we can grok the essence of loss. Are you ready for this, man? I couldn't be more ready. Let's go. Okay. My name is Dr. Greg. That's the whiz today. We are digging down into loss. And this one has been one we've sort of been working towards for quite some time. If you guys want to go deeper, there's lots of content that supports us driving further and further down into the LLM. But we want to sort of combine our transformer series that included how we get stuff into the transformer, how we pay attention to it, how we predict the next token. we pay attention to it, how we predict the next token with this idea now of once we can get outputs, how do we train models like an LLM using this idea of loss? So with that in mind, please do ask questions along the way. Let us know how you guys are thinking about this and what issues conceptually and from a code perspective that you're having. And hopefully we'll have a great discussion and event together today. Let's go ahead and dive right into it. Logits and loss, everybody. This is going to be how we do the training and fine tuning that's really at the core of all of the awesome machine learning that's happening out in the world today. Number one, as we align ourselves to the sesh today, is we want to understand what loss is even doing. Very, very important. We want to understand cross-entropy loss today. In order to understand cross entropy, we need to understand entropy. And in fact, this will lead into understanding also relative entropy, which we'll see is useful for a number of things that you might already be familiar with. We want to learn how to take logits that are essentially the outputs of our LLM transformer decoder stack, and then compute loss based on those logits. And then really just trying to get a feel for how pervasive this concept is. I think you probably have some insight from our intro today, but hopefully we can give you a little more to chew on by the end of our hour together. All right, so why loss? Why logits? What can we learn about cross entropy? When we look at computing loss in practice with a tool like PyTorch, what does it look like under the hood, and how is it slightly more complicated and potentially slightly more streamlined than trying to think about this from a first principles perspective? We always want to be seeing our loss decrease as we're doing training. Many of you are probably familiar with this. And why loss? probably familiar with this and why loss? Well, it's because like in order to train, we have to know loss. And you know, the thing about like this word loss, like what is this thing that we're even talking about? Well, this loss function is not something that's new or super complicated from an idea perspective. The loss function idea is the same idea as a cost function in classical optimization. We need an optimization target. It's the same idea as an cost function in classical optimization. We need an optimization target. It's the same idea as an error function. Remember y equals mx plus b and we could sort of see how far off we were from our predicted line with the real data. We could calculate many different error metrics We could calculate many different error metrics, and those metrics would sort of give us a single number that we could then use to optimize our line that we were fitting to the data. This is exactly the same idea here. We want to take what we're actually seeing, our actual data. We want to take our sort of predicted line, in this case, through 2D space, but our prediction. We want to compare them, and we want to make sure that they're less far apart than they were before we were doing this step of training or fine-tuning or alignment. And if we do that, then we'll see that loss in essence really is the same, whether you're talking Y equals MX plus B, an MSE or chat GPT. Okay. But when we talk about chat GPT, when we talk about an LLM decoder stack, But when we talk about chat GPT, when we talk about an LLM decoder stack, we're not training models to sort of predict this two-dimensional thing. We're training models to predict the most likely next token. And so when we want to predict the most likely next token, we want to sort of frame this in a loss function through like a loss function lens. And so when we think about LLMs and we ask, well, what is loss exactly? We can say that we're sort of taking the distribution of potential next tokens that we have and we're sort of trying to move that distribution a little closer to the correct next token distribution. And the way that we're calculating this is with a metric called cross entropy. We're looking sort of across the distribution that we have to the distribution that we desire. And we might say, well, we're looking across what exactly? And the answer is looking at the logits and the targets. targets. And we talked a little bit about this in our next token event last week or two weeks ago. And we saw down in some of the weeds of the code that cross entropy is taking the logits, it's taking the targets, and this is happening during training. During inference, we don't need to compute loss because it doesn't really matter during inference. Now you might be sitting there thinking like, hold on, what exactly is going on? We just used a lot of words and maybe we could stand to sort of slow this down a bit. So let's do this, but a little more step-by-step. You guys remember the OG Y equals MX plus B. Well, we want to think of chat GPT in a similar way. What we're doing is we're giving an input X, in this case, I want to learn, and we're getting an output, how. And then what we're doing is we're shifting and we'll shift to want to learn how to. And we'll shift again to learn how to cook. And so on and so on. And we're going to shift as we move through from input to output in this generative AI tool. We're going to continually pick the next most likely token. Now, GPTs are what are known as auto-regressive. That means they predict future values based on past values. They predict Y based on X. So X goes in, Y comes out. This is the same as saying they are causal. Causal LM is something you'll see in the code a lot. And they're causal because what was put in causes what comes out. So at each step, we are deciding what's the next most likely token to be predicted. And when we do it well, and we sort of tune it for chat, we align it with the things that humans would expect, it actually sounds like you're having a conversation. Pretty cool. So picking the next most likely token is really picking the next most likely token in the generated probability distribution. Because when we come out of the transformer, we're going to generate a probability distribution of all potential tokens. And so this idea of what loss is doing is we're comparing the generated probability distribution to the desired probability distribution at each step of next token prediction. And so we can kind of see here, we'll have a Shakespeare example today, because it's just sort of a classic to use with nano GPT that we started leveraging for our last event. And we'll continue to sort of leverage as a learning tool, shadow Tondra Karpathy. You'll see that this sort of leverage as a learning tool. Shout out to Andrej Karpathy. You'll see that this sort of first input here is sounding very Shakespearean. Spur me, have I? And so you'll see sort of we drop the first token. We pick up the last token. We shift our window. And so we have me, have I is where we start, me, have I done. And we can sort of go continue through with our next token prediction using this context window. Now, if we sort of represent this input as a distribution and we represent the output as a distribution, just conceptually, you could sort of say that during training and during fine tuning, we're sort of taking this generated distribution and we're taking this desired distribution and we're trying to kind of merge these together. That's what we're trying to do. We're trying to sort of make these kind of the same. Ideally, if trained very well, these will be the same. Leaving aside, if you have a transformer and an AI that's actually so predictable, you want to sort of start mixing it up again, that maybe you choose to do that. But this idea of during training, we're moving these distributions closer together. We're doing this through the use of cross entropy loss. through the use of cross-entropy loss, our generated information that we're using to represent this are the logits. And our desired information that we want, those are the targets. So what we're doing is we're taking the logits and we're taking the targets and we're sort of bringing them together towards being able to predict the next token correctly based on the data we're training on. Spur me, have I done. on? Spur me, have I done? Spur me, have I done? I should say. Okay, so this idea of logits is very important. Targets is, well, simple enough. What are we aiming at? But why logits? Well, let's break down this word. It's kind of important. Logit just means the logistic unit. The logit is the log of the odds. Cross-entropy loss is actually also known as log loss. Put a pin, log, log, log, log, log, in that for just a moment. Cross-entropy loss requires us to understand entropy. And when we think about entropy, of course, there's lots of ways from a physical scientist's perspective to think about entropy, but entropy in a next token prediction context is very well thought of as surprise. Because if you know exactly what somebody is going to say, if you know exactly what your LLM is going to spit out, if what's coming out of the LLM or somebody you're talking to's mouth is highly probable, there's very, very low surprise. Not surprised at all. Absolutely knew it. Knew you'd say that. In other words, as probability increases, surprise decreases. We could say that these are inversely proportional to one another. We could say that the odds of being surprised decrease with more predictable words coming out of a machine or a human. And surprise then could be thought of as kind of the inverse of probability. And this is where we can start to connect some of these ideas, because if we're 100% sure about what somebody is going to say, then the probability that this happens is one. But the issue mathematically that we have, and this is one of the issues with just calculating inverse probability, is that if we take sort of one over the probability of one, sort of measuring how surprised we are, sort of measuring how surprised we are. We don't want to get one. We want to get zero. Because if we're not surprised at all, why are we returning a value that would indicate we're like super surprised? Rather, this is where we invoke our friend the logarithm because if we take the log of one over one that's the log of one that gives us zero it plays very very nicely with this idea that if we're 100% sure, then we're not at all surprised. So it's useful. Now, if we sort of take this idea of cross-entropy loss and sort of stack some of these ideas together, we might think of this as sort of the surprise loss. And of course, we can think of it as the log loss for obvious reasons, because we're using the log. And the logits, remember, are simply the log of the odds of the probabilities. So all this comes together. And there's really not a whole lot of really crazy ideas in here. We're just sort of trying to say, can I develop a mathematical model that sort of represents the way I think about this in reality. If a word comes out, is it surprising? And so we have this idea of entropy. This idea of entropy leads us to now being able to stack into this idea of cross entropy. into this idea of cross entropy. But let's talk about calculating entropy first. Conceptually, all that stuff and sort of wordsmithing is fine. This is originally from Claude Shannon of Bell Labs way back in the day, sort of the godfather of information theory. And generally you're gonna see an entropy equation that looks like this. But you can actually just rewrite this to make it sort of align with the discussion we just had a little bit better. and shout out to StatQuest on YouTube for crushing it and giving me some insights into this. Entropy like this kind of gives us this log of one over the probability which is kind of that surprise right so what we're actually sort of able to talk about is we're able to sort of talk about this in a more intuitive way where we kind of can say, well, what's the probability that I'm going to pick something out? And then how can I create a metric using that probability that sort of denotes surprise? And the example from the entropy video, which is a fantastic one, uses chickens, right? So this is very easy to grok this. We got six chickens that are orange, one chicken that's blue. And so what we are going to do is we're going to sort of calculate the entropy here by looking at what's the probability that I pick out an orange chicken if I select sort of at random any old chicken. And we're going to have the probability of six out of seven. We're going to put that six out of seven right into our surprise here. And the blue chicken will be probability one out of seven. Probability one out of seven. And we can get this number. The single number is the important idea here. Remember, when we have a loss function, we don't need a whole bunch of numbers. We need one number. That one number is going to drive our optimization. It's going to drive, ultimately, the performance of our algorithm that we're training. to drive ultimately the performance of our algorithm that we're training. And so we can take this sort of idea of entropy and we can now look across generated and predicted values, generated and predicted tokens. We can modify this slightly to look like this. Very simple. And when we look sort of across generated and predicted tokens, what are we doing fundamentally? Well, we're looking across logits and targets. taking this kind of distribution of, in our case, words, and we're saying, how do I measure how close my distribution of words that I have now is to the distribution of words that I have now is to the distribution of words that I want. And of course, this is easily implemented in code where we can simply invoke the very nicely laid out cross entropy function. And it'll pick up our logits that come right out of our transformer. It'll allow us to compute our targets. It allows us to take our targets and then compute, again, one single number from this. And we can get better and better and better at choosing whether I should be talking about grandma's funeral, grandma's house, or grandma's birthday. And again, this is only important during training. During inference, I don't need targets because I just have what I have. I don't need a loss function. I don't need to know exactly how far I am away from how far I want to be because I'm not trying to go there right now. I'm just trying to get an output. And so we're going to see an example here where we're going to use some Shakespeare data and we're going to compute actual logits. There's going to be a little bit more complexity than we talked about so far because computationally there's sort of a lot going on when we deal with LLMs. Things are quite large. But we're going to go ahead and hand it off to the whiz for a walkthrough on how to calculate loss right now. And then we'll come back and we'll sort of sit and we'll discuss, okay, what have we seen? What are the implications of this? What exactly can we take away from connecting the lower level theory to the higher level computations that are actually happening sort of at scale in these LLMs today? So let's sort of get some insights into how really everything is using loss with the Wiz as he walks us through loss 101 down in the weeds. Wiz, over to you, man. Yes. Okay. So you can see here where we're starting cross entropy loss, right? So this is the PyTorch implementation. This is what we're going to see today. I think the most important thing to focus on here, and I'll zoom in just a little bit more, is this idea that the criterion computes, or this criterion computes the cross-entropy loss between input logits and target. Now target in this case is going to be a specific index. Now what that means is, right, so when we think about our our output layer as Greg's been talking to us right one of the things we know about it is that it represents a vocabulary and every item in that vocabulary is going to be a separate token right so if we have say a vocabulary of ten words we have like the cat, you know, whatever it is, what we can do is we can actually show the number that that is in literally a list, right? So that's the index. Sorry, my brain melted for a second, but the idea is that's it, right? So that's going to be the target. And then on top of that target, that's going to be the logits that are associated with our predicted token. So what that means is that we're gonna wind up with two things to compare, just as Greg was saying, right? Which is one is going to be a big list of numbers with only one value at the index that is our desired token. And the other is going to be a bunch of logits that's going to have the same shape as that other list, right? And then PyTorch very conveniently for us, if we look into the, if we look into the implementation here, PyTorch is actually going to do the softmax for us. So we don't have to, and this is all going to occur during our forward. So we don't have to do this, this, you know, softmax first, we can actually rely on PyTorch to do this softmax first. We can actually rely on PyTorch to do that for us, which is super convenient. So what we have to pass to PyTorch is just the raw logits and then, of course, our target, the index of our target. And this is part of what we're going to be learning over the course of training is how to predict the correct target or how do we get the right index, right? Indexes here are mapped, literally mapped to tokens, right? Through our decoding step. Okay, so let's look at the notebook. The notebook's not super intense today, right? The most important part of the notebook is this, what we're going to get to in a second. Okay, so first of all, dependencies. We have a few. Just to remind everybody, we are using NanoGPT from Andrej Karpathy. Absolutely amazing repository. One of the best for learning about these systems we're going to get the dependencies that we need we're going to get the repo and then we're going to go ahead and we're going to move into the repo we're going to use the prepare data set just to convert our big long blobs of text into a big long blob of tokens we're going to see that we have 300,000 tokens in train and 36,000 tokens in bal. Nice ratio. I've hidden all the preamble code. We don't really need to worry about it, but we need it in order to run. So we have it tucked away here because what we want to focus on is the training loop. The training loop is where we're going to have this idea of loss come into play. And there's one thing that I want to focus on kind of before we get to the training loop, right? So you'll notice that we've just printed out our model. Our model is just this GPT model. Our model dict is in a transformer or transform as a model dict that has our word embeddings. Pay attention to this particular value, right? This is the text embedding. So this is the direct embedding from our embedding layer. And you'll notice that this has shape of 50,304, right? Now that happens to be of course, the size of our vocabulary. So right away we know, right? When we come in and we have tokens, we're gonna go to some internal representation through the embedding layer from our vocabulary, right? So this is the input is going to be tokens. And of course, those tokens are, they're just numbers that map to a specific word outside of this process. Because even though it's a bit counterintuitive, the transformer never actually sees text. It in fact doesn't even care about text. What it cares about is these internal representations and then going to and from tokens. Since tokens represent text, all we have to do is a lookup and convert the token to text, which we'll see in a second. But that's the idea. We have this input that has that size, our vocabulary, and then we have the output, which is the LM header, the language modeling head, which we talked about last week, which is also having output size 50,304. Again, our vocabulary, right? And so this is the kind of tying together of these these tokens to this concept of what we wind up caring about and we're going to see when we calculate our logits why this is important all right so now that we're going to go ahead and train we have to think about batch a little bit now the word batch here is super overloaded uh when we say we're going to get a batch what we mean is that we're going to get a batch, what we mean is that we're going to get some information from our training set that's in shape 12 by 1024. Now, the student among you have probably recognized that that 1024 is actually going to be our sequence length, right? So that is the total sequence length of our model, meaning that our model can capture sequences of size 1024. You'll also notice this 12, we're going to largely ignore this 12. The 12 just represents batch size, which we can take advantage of the parallel computing of a GPU and the fact that it has a ton of memory in order to do many passes at once, which is referred to as a batch. And so that's what this 12 represents. But the idea is that every time we get a batch with this get batch function, we're going to get a chunk of data for x and Y in the shape of 12, the batch size, times the sequence, or by the sequence length, 1024, right? So every time we call this, we're gonna get 10, 1024 tokens from somewhere in our training set for both X and Y. And you might ask, well, how do X and Y differ? Well, we can look at that here. If you notice, right, you'll see that, you know, this starts with 1, 8, 7, 1, 9, then goes to 11, then 3, 51, where our labels start at 11, right? We can see the 11s match up here, then 3, 51, they match up here, then 4, 65, they match up here. So the idea is that Y is just X, but shifted so that we have the next token. And that is the idea of our transformer network, right? It's a next token prediction machine. So what this is helping us to train and how we're going to leverage this when we're looking at our loss is this is what tells us our target, right? And if you notice, these are just numbers and all of these numbers are going to correspond to one of the potential indices in our vocabulary, right? This target is just an index. That's it, right? So this is the thing that PyTorch was asking for, the target, right? So that's all we need. What makes this very good is that, you know, it's just predicting the next token. Now, when we do our generation step, as you see, we can actually get a bunch of information all at once out of this, but that's just something to pay attention to. Okay, we're going to also import this tick token for our decoding step. The reason we're doing this is simply to make sure that we can look at what our input is in text. There's no other reason. So we're just using this to showcase what the input text looks like. Even though the model is not going to see this text, it is only going to see the token representation. So these representations that are just those, you know, integers. And remember that in between the input and output, the representation is completely different. We're only talking about the very beginning and the very end here. Okay, so now let's go into our training loop. There's a lot of stuff going on that I'm going to completely ignore to focus on the thing that we care about, right? So the thing that we care about is the logits and loss. Now, the way that we compute our logits and loss is by doing a forward pass in our model. So you'll see here that we have logits and loss, which is grabbed from our just forward pass, which is model. Zoom in a little bit here so we can see this a little bit better. And that's it, right? All we have is those logits and those loss from our forward pass. If we look at our nano GPT repository, we look at our model.py and we scroll down to our actual forward pass that we're going to use for the model. We're going to notice that all it does is generate those logits and loss. As you can see here, this is our forward pass, right? Those logits are just output from the language modeling head, which has that, you know, that output dimension of that 50,000 we talked about. And all we do is use the PyTorch cross entropy to look at all of the logits for that particular output and then compare them to our targets. That's it, right? So that's all we have to do. Now, if we don't have any targets and therefore we're doing inference, we ignore the part where we get all of the logits. We only need the last set of logits, which is what represents our next token prediction. But when we have targets, we're going to generate, we're going to look at all of them, right? So this is the key distinction between inference and training for this particular model. And that's it, right? So all that this is doing when we do a forward pass is it's giving us the output from the language modeling head, and then it's giving us the loss as calculated through the PyTorch cross entropy implementation. So that's very handy, right? If we go ahead and just close our window there, we can see that we can print these things out pretty straightforwardly, right? We're going to print out our inputs, our input in text, our targets represented in text, the shape of our logits, just so we can do some intuition building building and then our vocab size and sequence length which we get from our shape so let's scroll down to where training happens so in training we can see that our inputs truncated to last 15 tokens in the last batch were a bunch of numbers that makes sense and then the actual represented in text version is the error and be not fixed and doom perpetual hover right now what this means is that the model only saw these numbers of course but these this is the text representation now notice that the text representation of the target is the same thing but shifted forward one step right and that's important to keep in mind, again, because that's what we're comparing to. We're calculating that cross entropy loss, right? Across the whole sequence. So every set of tokens has logits. And then every set of logits has a label, right? Or a target. And what we're gonna do is we're gonna calculate all of those losses. So when we look at our shapes, we'll see that we have a vocabulary size of 50, 30, 5,304. And then our sequence length is going to be 1024. You can kind of, and you can see just the raw shape here, which includes our batch size. But the idea is that we have this, we have these logits, which are in the shape of our vocabulary. have these logits, which are in the shape of our vocabulary, right? So each position or index in that array that's of size vocabulary is a score for the index in that vocabulary, right? So what we're going to do is we're going to compare that to our targets at every single step of the sequence, right? So what this means, because our labels, if we look up above, right, our labels have this shape 1024, right? So what we need to, you see this here, the Y has shape 1024. Now, what this means is that we have 1024 labels, Now, what this means is that we have 1024 labels, and then we have 1024 logits, right? Now, labels are, you know, they don't have an extra dimension, so we're not going to count them here. But the idea is that for every input sequence, or for every sequence length logit, we have a label, which is the next token. And then we compute all those losses. And then we roll all of those 1024 losses up into a single value using averaging or addition, depending on your strategy. And then we'll be left with this, uh, you know, this, this list of 12, which is our batch size. Again, we're mostly ignoring it, we'll be left with a list of 12 losses that we'll roll up again to count as our total loss, which is how we wind up, despite having this multidimensional array, right? This is how we wind up with a single scalar, just the number, right? This is the idea. So with that out of the way, I'm going to go ahead and pass you guys back to Greg. And we're going to we're going to do some discussing. All right. Yeah. OK, so we are down in the weeds here. Let's go ahead and let's go ahead and take a look. See at this visualization of nano GPT, just to try to connect some things right now. We talked last time when we discussed the next token prediction and we saw just now from you, we're talking about this LM head and we're, we're sort of calculating these logits. So we're taking all of our vocabulary by our embedding dimension, and we're multiplying it by our embedding dimension times our sequence length, t. And what we saw for what you just showed us here is that we're seeing that actually this LM head vocab dimension is actually not three, as in this nano GPT, but it's actually 50,304. And this token sequence length isn't 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, it's 1024. And so like, we're dealing with some quite large matrices here. Now, I wanna sort of ask as well, if we kind of zoom in on what's actually happening down here, if we imagine that our logits here are this sort of 1024 by 5304, you said there's also 12 batches, right? Yes. And those batches, what's the best way to visualize them here? If we're sort of thinking about the batching. Yeah, it's just like a plate. Plate's going back, right? So we have this one two by two. Just imagine 12 of them stacked on top of each other. Oh, okay. Okay. So just like in the third dimension here, we're sort of stacking back. Okay. So there's just like, there's a lot going on under the hood of this pie torch stuff in short is kind of the point I want to make here. And then if we kind of consider, okay, like you also mentioned that under the hood of PyTorch, we're not doing the computation with logits. Rather, we're actually using the softmax logits to do this computation, but we don't handle it. We just give logits as input to the function. Is that right? Yes, that's correct. PyTorch does a special set of operations, which are the equivalent of what we want to do that are computationally stable and efficient. Okay. All right. So PyTorch is better at it than you are. Yes. Yeah. Okay. Way to go, PyTorch. All right. So we want to kind of expand the frame now and definitely, you know, start throwing your questions in the chat here. Because one of the things that we can actually start to understand, given this context, is we can start to understand that if we actually can calculate cross entropy by looking across two things, and we can calculate entropy, then what we can do is we can back out this idea of relative entropy. Another name for relative entropy is the KL divergence. Shout out to K and L, OGs in the game. And we can sort of take cross entropy and subtract out entropy. And in fact, we see this happening in a lot of places, right? So we did an event on RLHF some time ago, and this is like the classic diagram that everybody's like, oh, what is this? And this is like from Hugging Face. And we see this sort of KL prediction shift penalty. hugging face. And we see this sort of KL prediction shift penalty. So really the way we've been talking about this, if you've checked out our events, is we try to avoid this sort of super mathy discussion and rather we try to sort of break it down into more of a sort of tactical understanding. So kind of walk through how we can think about this in terms of like, what we're looking across. And then what we're sort of calculating the entropy of in RLHF. Yeah, so the idea is that we want to know the the delta or difference between the two distributions, right? So we have some distribution, you know, let's call it PPO, and then we have some distribution, let's call it base. And the idea is that we don't want these distributions to diverge too much, right? So the way that we do that is through this KL divergence, or sorry, KL prediction shift penalty. Yeah, yeah, yeah. Specific words on the screen. And what that does, basically, is it says, says like if your generation gets too far let's say away from the base we're going to receive a uh we're going to receive more penalty than if it doesn't and the way this is leveraged in uh rlhf is that we actually set a hard bound on the on on the, uh, on this Delta. So we don't allow it to exceed a certain, uh, a certain rate. And what this means is that we can, we can encourage our model to update its weights while being effectively clamped to our base distribution so that we don't wind up just producing like, say the token for cat 7,000 times. That's what the reward model would like, you know? the base model and our policy model. And we're saying, we actually don't want you to be too far away from the base model. And the base model is where we're calculating our sort of core entropy. The cross entropy is where we're looking across the models. And then we're sort of providing this constraint of do not shift too much beyond the base. And so very interestingly, we sort of have this reward modeling piece that's happening here as well. And we did mention that reward modeling is also an area where we're using loss. Of course, here we're updating the weights. Anytime you see weight updates, you should be thinking, ah, there's probably loss there. There's probably loss. So this is actually sort of just a check and it's not the thing that's doing the training. It's not the thing that's doing the actual fine tuning and the alignment and actually leveraging the loss function to update the weights. Rather, the reward model here is doing that. Is that correct? That's absolutely correct. Yes. So the KL divergence that we want to be low is simply like you can think of it like the railroad tracks, right? We we don't or well that's a bad one because it's stapled to those let's call it a let's call it a uh a mountain path to use the hugging face classic right where if we step too far off the path we're gonna fall off the cliff right so what we do is we say okay okay, you can walk within this path, but we actually use the input from the reward model, which is computed using a slightly different loss, right, to what we've been discussing so far in some cases. You could use an LLM there, of course, but if you say like a classifier, it's a slightly different loss, very slightly different. Um, and that's the thing that we're actually going to use to drive the, the, uh, magnitude of our updates. So one thing that I do want to be very careful about saying, we still do use the loss that we wind up with traditionally through cross cross entropy loss, but the magnitude of the weight impact or weight update is going to be dictated by the relative reward, where a low reward is going to encourage small or few or minor weight updates, and a large reward is going to encourage large weight updates. Okay. Okay. Okay. All right. So I just want to sort of go to this most classic example of what's happening here, because it's interesting in DPO. Now, I want to focus your attention in on the idea that we're doing O here. We're doing optimization. Anytime we do optimization, we need a cost function. We need an error function. We need a loss function. And in fact, that's exactly what we see here. It's more explicit in DPO. That's one of the sort of benefits of DPO is we're able to sort of put this very explicit loss function together. We are still doing our checking through KL divergence of reference versus aligned. And the loss function that we've again talked about previously is something that looks like this. And you can start to see some similarities between loss functions, even if they are a little bit different than maybe the one that we covered today. We're seeing the logarithm and we're seeing a number of other key terms in here for a slightly more complex system and optimization than the one that we're sort of using with nano GPT. So just to sort of get back to where we started here was we've got all of these things that use loss. Are they all using different loss functions or are they all sort of taking this idea of cross entropy and kind of using that as a starting point? What's the best way to sort of think about this and really extract the essence here. So let's say broadly of the listed technologies, cross entropy is going to be an important part of their loss puzzle. So just to be very, to be as robust to the small differences between them as possible. Now, because of that, right, there and this is typical kind of of the next token prediction game or the next, you know, pixel prediction, yada yada, right? Some of these, like, for instance, reward modeling, right, might use a slightly different loss function, depending on if we're doing say binary classification for uh our reward versus if we're trying to get an actual like regression score the dpo uses a modified version of loss which takes into account preference so where whereas for rlhf we just talked about this whole KL divergence reward model, add magnitude, all this other stuff, right? DPO kind of skips all of that and translates it back into a familiar cross entropy-y form. And what we can do with that is we can kind of in the actual cross entropy loss calculation, we can inject the preference information, which is where it gets its fun name of direct. So, and then for end to end rag, we have two different losses that we care about, right? We have a loss from our retrieval pipeline, and then we have a loss from our, you know, our generator, our LLM. Now these two losses are actually not the same kind of loss. They're two different losses, right? They are not both cross entropy loss, but thanks to the magic of a number need to get lower. The same intuition holds where we can, we can kind of Frankenstein these losses together. And because the idea is that a most optimized model is a model that has the lowest loss, we can just stitch them together. And we know that even if the sum of those two losses is always going to be higher, we still have the information that is relevant, which is that one loss that is better than the last loss will be lower. Regardless of which system improved, regardless of which piece of the puzzle changed, if the net loss goes down, we can say that the system is performing better in that loss landscape. Okay. And so just before we get to these sort of other modes here, so it's similar to the way we saw that, like, we could take the entire sequence length, we could sort of compute loss for each token in the sequence. We could amalgamate this loss all together into one lost number. We could look across all the batches and get all the, and then we're just one lost number. And then we're actually going to go and we're going to then back propagate this information through the network, update the weights, and then we're going to do better at achieving that output and that outcome next time. And so sort of no matter how you get this number, which you're going to use many similar techniques to the ones we use today, you're going to be able to do this. And in fact, I've noticed that there's a lot of innovation happening with modifying loss functions in sort of state-of-the-art papers. And I've noticed that researchers tend to be thinking about ways to do this, that our potential game change, DPO is a great example, right? That's right. And so looking out for this is something that maybe will encourage everybody. Maybe we'll leave vision multimodal if you have just 10 seconds on what we should think about this for today before we sort of move on to closing up the Q&A, final Q&A. Yeah, it totally depends on the possible outputs of the models. But for the most part, it's just going to be in the same kind of lost world, especially if it's like our text is our output, then it's very cleanly going to be exactly what we talked about with some different top layers in order to get that internal representation. There's no real extra or new information to add beyond that, I think, especially with short time. The idea is that we still care about minimizing our loss in order to represent our training data the best that we can. Okay. And then, you know, I was just struck by how this was slapping with a lot of truth as we dug down deep into this today, this particular classic, what a classic meme this is. And then, you know, we're kind of seeing from regression to classification, we see the loss functions, we see the training, we see the similarities. From ML to LLMs, we're seeing a lot of things really come together, which is very, very nice. And I think they've updated this meme to include Gen AI these days. I've seen that as well. So I think in closing, if we could sort of wrap this together, like cross entropy loss, aka log loss, is sort of telling us about like how surprised we are about the next token. It's allowing us to minimize this surprise. The logits, the log of the odds, help us to be not surprised when we put a high probability on something. I'm not surprised that I'm not surprised. Thanks, log odds. And if we look across those log odds and the targets that we want to achieve, we can take where we are, which is the outputs from the LM head into the logits. And we can take where we want to be, which are the targets, put that into PyTorches. Nice tool for us. And we train ultimately to be less surprised than we were before we started training. So like you do with training anybody or anything or receiving training yourself, you want to be less surprised when you see that stuff in the future. So we just have a couple of questions today and I'm going to go ahead and there's a funny one on grokking that I don't think we're going to do a session on grokking. Endless grokking is a thing that I don't know about yet. And for supervised fine-tuning, we use causal language modeling approach. What about unsupervised fine-tuning? Is masked language modeling used? This is a great question because I think it's an often confused point. Yeah. So technically what we showed today is the unsupervised version,. Unsupervised here is an interesting thing to get stuck on. What I'll say is that it's all supervised with language modeling at the end of the day, even if we're not forthright about that. Yeah, yeah, yeah. Supervised all the way down. Because we need an input and an output. But that's a good question and sort of a common one. Can the logits vector after softmax also be considered as an embedding? Rustam asks. No, I wouldn't. I wouldn't. You perhaps mathematically, we can get away with kind of like tricking ourselves into saying that, but it's not right. And in fact, I would think of I would encourage you to think of logits as not being a vector. It's just a collection of scores of arbitrary scalar values. They don't represent anything together, right? It's just a score for each pip in the possible buckets that can have score, right? So I would encourage you to not think of logits as a vector conceptually. Rather as a distribution with the buckets. Yes. And then same with Softmax explicitly converts us into the world of a probability distribution. And I'm sure there is some math that exists somewhere that could tell you that a probability distribution is an embedding, but we're not going to use that term as it relates to the rest of our systems, because that term has a pretty specific meaning in the rest of the model that we want to avoid confusion with. Yeah, yeah. We might embed some numbers into a probability distribution representation. Okay, okay. Yeah. All right. It's time to wrap up, everybody. Thanks for joining. I do think that we are going to potentially go further down this rabbit hole in about two weeks. I think we left off about where we need to start back propping through. So look out for the next step in our journey. Thanks for following us guys. And thanks for showing us the way, man. That's a wrap today, everybody. Thanks for joining us. Please do like and subscribe if you haven't yet. We're here every Wednesday doing our best to deliver real value to you guys that want to continue to learn at the open source edge of AI with LLMs. If you're still here, you're going to be great for our community. If you're not in it yet, please jump into Discord and start building, shipping, and sharing today. There's folks building, shipping, and sharing cool stuff from our courses and beyond in the Build, Ship, Share channel every day. And if you're interested in checking out our top-rated course on Maven called the AI Engineering Boot Camp, we are doing some really cool stuff with it. And keep an eye out. We'll be making some announcements soon for the next cohort, cohort four that starts in August. If you have any questions, feel free to reach out to me directly. And we will be continuing this series in the coming weeks. Check out our Luma page for upcoming events. We hope to see you there. Luma page for upcoming events. We hope to see you there. And finally, if you have any feedback, we'd love to hear it. And we'd love to hear what you guys would love to see next and how we can do better and better and better building, shipping and sharing next time. As always, we will keep building, shipping and sharing. We hope that you guys absolutely keep doing the same. See you on Discord. See you on LinkedIn, everybody. Have a great week. Bye, guys.
Logits and Loss: Training and Fine-Tuning LLMs
Join us as we unravel the essential role of cross-entropy loss in training and fine-tuning Large Language Models. Discover how this foundational loss function optimizes predictions, from standard methods like Low-Rank Adaptation (LoRA) to advanced techniques such as Direct Preference Optimization (DPO). Learn how cross-entropy loss helps make LLMs more effective for specific tasks and improves their performance. Don't miss this insightful session—subscribe and watch now!

Event page: https://bit.ly/logitsloss

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/k5zBBvLo8oSCht1grbUeqc

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

For team leaders, check out!
https://aimakerspace.io/gen-ai-upskilling-for-teams/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/g4MyGn7NUWPxEhGY8