 Yo, Wiz, true or false? Transformers are basically just fancy classifiers. Yes. That's right. Okay. All right. How about technically the loss function used to train transformers and the loss function used to train classifiers are the same. They share very, very, very similar roots. Absolutely. Oh man. Okay. I'm excited for this one, man. Are you pumped to kind of get into the next token today and finish out our story of the transformer? I know I am. I'm at a loss for words, Greg. Oh yeah. All right. Let's get into it. Can't wait to have you back in a bit, we'll be discussing throughout the day. We will be digging in to the deets and we are here to discuss really one of the most important things that's at the root of all of the applications being built today in AI. My name is Greg. That was the Wiz. We are AI Makerspace. Thanks for taking the time to join us. Today, if you've got questions along the way, please drop them in the Slido link. Please drop them in the Slido link. Please drop them in the YouTube live chat. This should be a fun discussion, and we're excited to sort of kick off a new series of deeper dives into the Transformer and training inference and beyond. Everybody, welcome to the next token, how LLMs predict. So today is going to be kind of a fun one as we align ourselves to the sesh. What we really want to do is we want to kind of bring back the different pieces that we've talked about already, if you've been following, and we've got available if you want to dive deeper. We are almost at the end of the transformer stack. And once we can put in something to the transformer, get something out, we can do inference. And what's really cool is that once we can do inference and we can see what goes in and what comes out, then we can really talk about, well, how do we sort of compare what goes into what comes out, in and out, and optimize that, essentially do training. So we'll talk about what goes in, what comes out, exactly how we're predicting the next token after we're done with our decoder blocks. And then what implications does this have on training? And how can we connect this back to the classic roots of ML and AI that we all learned early in the game? So first we talk Transformer, then Token, then Loss. This should be fun, everybody. Let's get into it. So Transformer, of course, we see this kind of thing a lot. It's terrible. I personally can't stand it. I hope you can't stand it too, if you're watching our channel. And we've sort of discussed, actually, this is not the one that is mostly out there. The GPT style is the decoder only stack. We've talked about this in a previous event that you guys can check out. But if we zoom in on this, what we can see is we can see that there's some stuff sort of going in here, some embeddings, some positional encodings. We've talked about this stuff in a big long event we did talking about the Lego blocks, but we kind of got stuck because it took so long to talk about it. As we go into the decoder block, we run into masked multi-headed attention and just generally self-attention. We went into quite a bit of detail on self-attention, the QKBs and all of that in a previous event as well. Today, we are excited to finally dig in to the top of the stack here. And the top of the stack is really where we're predicting that next token. Everybody just says, oh, you know, the LLMs aren't so smart. They're just next word predictors. Well, let's take a look and see if that's true today. So as we go from in to out, we want to kind of see how this comes together. So as we start our journey into the next token here, I want to kind of flip this thing up on its head. One of the things that the best visualizations out there suffer from is consistency and which way is up and which way is down. So let's actually just go ahead and flip this bad boy on its head so we can get a little more intuition about how this is actually operating and we can get some better visualization of exactly how things are moving through the entire stack in a more realistic transformer. This is, of course, from the attention is all you need paper. There was this really great visualization put out recently that we're going to use today, and we're going to focus on taking a look at nano GPT. This, of course, looks like gibberish right here, but we're going to kind of focus in on how one of these things starts to come together. A little bit more complicated than the paper, a lot less complicated than what most of the stacks out there are actually looking like today. And if we sort of look at this nano GPT stack, we can see that there are three decoder blocks here. And these go through self-attention, add norm layers, feed forward network, aka multi-layer perceptron. And we're sort of moving through this stack. Now, like conceptually, very cool stuff is happening. What are we paying attention to and why? And we've talked about that in previous events. Today, we're getting a little bit more down in the weeds. And what we really want to do is want to kind of, again, focus here on this piece of the stack. So I want to take a closer look at NanoGPT with you. Now, if you're not familiar with NanoGPT, this is something that André Carpathi built, straight ledge, obviously, in the field. And it's a great one that we used to start with in a former class that we really were digging down into the details of it. It's a great one to start with in general. And I'm going to go ahead and show you guys what this thing looks like. If we look at our visualization and we kind of see it here, highly recommend that you check this thing out and you play around with it if you're interested in this kind of stuff. But if we kind of zoom in on the top here, what we can see, and this is a simple example, okay, what we can see is we can see that what we've got is we've got sort of our embedding representation here. Now we're trying to solve a very simple problem here. We're just using character level stuff and we've only got sort of three different characters abc that we're dealing with but we don't really need to pay attention to the specific use case as much as we just want to look at the structure here okay so what we have is we have our sort of our tokens of length t here all right that's that's going to be pretty important as we continue on. And what we've also got is we now have sort of for each token that we have in our sequence, we want to have an embedding representation of that token. And so if we look at our input embedding representation here, what we see is we see that we have T columns and we have C rows. Okay. So that means we have the number of columns equal to the number of tokens. And then we take our token embedding representation. We take our positional encoding information, smash these bad boys together and we have a number of tokens in the sequence by embedding length representation. Now this is pretty cool because this is the representation that we need to sort of stick with throughout. Because what happens is as we start to move into our transformer, what we see is that we see like there's a lot of stuff going on in the transformer here we can kind of see that if we track our embedding representation t by c we have a similar t by c layer norm representation we have a similar t by c projection that's coming out and a similar t by c pass to the next transformer block so you can look at attention you can look at the projection but more generally it's it's interesting to look at this as a whole transformer block the transformer zero block and you can kind of like scroll down as we sort of already talked about we've got transformer one transformer two zero one two transformer three and so what's kind of most interesting to us today is that when we get down to the very bottom of our output to the very bottom of our output, what we see once more is we see our good old friend T by C, tokens by embedding length representation. And so the question is, okay, well, what's happening to this exactly and how's it being transformed to give me the next token and this is sort of the crux of the issue that we want to dig into today all right and so we see a number of pieces to our puzzle um but we're going to sort of discuss how to move through these specifically discuss how to move through these specifically to get to a logits representation that's going to give us everything that we need to decide well what should our next token be Interestingly, once we move through our decoder stack, we actually need to decode. That is, we need to decode our T by C to decide how to choose the next token. Because what's happened in T by C is I'm in an embedding space, right? But what I need is I need to get back into a space of vocabulary, a space of language. Embedding space is a vector representation. It's not a natural language representation. So how do we go then from tokens or from a distribution, from tokens or from a distribution, this embedding space representation from two tokens. How do we go from this embedding space representation to tokens exactly? How do we do this? Well, in order to get some insight into this, let's zoom in a little bit. Recap. C by T goes in. C by T comes out. Now, as we look along the stack here, we see that we have language model head weights. And language model head weights are of size n vocab by into this embedding representation to get this distribution. Because when we multiply n by c times c by t, we get a vocab by token list. Okay. by token list. Okay. So this is, and for this, you'll recall, we've sort of got one, two, three vocabulary letters, A, B, and C. And then we've got sort of our token sequence length here. sequence length here. And so this is pretty simple from a matrix algebra perspective. In fact, Wiz would tell you it's pretty simple from a programming perspective, and you'll see that today. And maybe it is. What's really cool here is that we're sort of moving from this embedding space back to this sort of distribution. And this distribution then incorporates our vocabulary information. And what we can do is we can take this sort of unnormalized distribution of which word has the highest probability to be the next word and we can sort of somehow decide through these next two steps. So if we think about this idea of going from logits to Next Token, we're sort of talking about going from distribution to Next Token. And the way we want to think about doing this is by picking the most likely Next Token. Okay, but how do we pick the most likely next token? Well, the answer is the soft maximum. That's right, the soft maximum. We get the maximum, but we also get everything else too. We get the stuff that's kind of close to the maximum. Now, this is still a naive programming technique. That is one that prioritizes imperfect shortcuts for the sake of speed, simplicity, or potentially lack of knowledge. speed simplicity or potentially lack of knowledge but it's not a greedy algorithm like the hard max would be if we use the arg max this is just going to tell me the absolute highest probability and so this sort of if you use that sort of argmax hard version, it's just going to say, okay, boom, this is the most in the most predictable possible way right you'd be pretty boring wouldn't you you kind of sound I don't know a little bit like a robot a little bit like a machine and this is maybe not super ideal. So a softer max is a way to sort of say, well, maybe sometimes we don't want to pick exactly perfectly this top most likely next token. Maybe we want to add a little flexibility, a little softness to this. And there are additional ways that we do this, of course, in state-of-the-art models. Now, what's interesting to note here is that we could take this set of logits and go directly to a token by building a classifier head. We could do that. Only we choose not to do that here. Weird. And it has a little bit to do with softness. But I want to bring Wiz back up to talk a little bit more about how he views the world here. And I think the right way to sort of get into this is, what do you think the right way is to select a next token in general? Because that's kind of an important question, isn't it? Yeah, I mean, if I had the exact answer, I bet I'd be, I'd be on a cruise ship somewhere or yacht somewhere. But I mean, there's not a real correct answer here, right? How there being a right way to select the token, I think is opposite to our intuition that our intuition would say, you know, like, that's very situation dependent. It's kind of dependent on what came before, yada, yada. The idea is that there's not like a perfect way to select the next token. And in fact, you you might claim like well what if i uh you know what if i i always want to choose what the correct token would be as for my data well that's still not necessarily the right token because that's just going to reproduce our data set which is not uh traditionally a very useful thing if we wanted a data set, it would be much easier to just print it out, right? Then build such a complicated algorithm to print it out. That's right. So this idea of the right way is kind of like, well, I mean, in an inference space, it's not necessarily clear. And this is sort of a lot of where kind of prompting comes into play, a lot of flavoring it like, hey, you want to think about you're talking from this role, or you have this feeling, or you want to make the customer feel like this or that. And this is what is so interesting when you dig down deep into here is that we have enough flexibility there to be able to do this. And then, of course, the number one parameter that people talk about all the time, because I think it's like the one you get access to with Open AI, maybe that's why, is temperature. And I guess technically, we could call this a smoothing of sort of the distribution that's happening down there. This temperature idea where like a higher temperature makes a little more uniform and a lower temperature temperature will make it a little more concentrated on the higher probability points. How should we think about, you know, using temperature exactly when we're building these applications? Yeah, it's exactly what you described, right? So this idea that temperature scales are logits. So if we have logits and we scale them by a very, very big number, which is to say we divide them by a very big number then they're going to kind of get closer to quote get closer together right and so when we soft max that the distribution is going to be a lot smoother or flatter right more close to uniform and because we're picking tokens from that distribution, we're going to be more likely to pick tokens that, you know, that are quote unquote further down the line than say the token that was scored the highest. And the opposite is true, right? If we divide our logits by a number less than one, they're going to become bigger, right? It's the same thing as multiplying them by that factor. So the idea there is if we have a lot of space between each of our logits, right, where the biggest one is going to grow proportionally more than the smallest one grew, we get this idea of a very sharp probability distribution where our most likely token now dominates the probability distribution is much more likely to be selected. And what that means in practice is that we're more likely to get the expected next token when we have a very low temperature. So a temperature close to zero. Obviously, we shouldn't have a temperature of zero. So we can't divide by zero. But many libraries, you know, deal with that with with epsilon or whatever. But the idea is, you know, if we, if we always want the model to give us what's expected, we should choose or sorry, not always to be very likely to give us what is expected. We choose a very low temperature. If we want it to sometimes make like a left field choice in terms of the token, right, that it's going to select, we should use a high temperature. This is often expressed in kind of a simplified form right as the model will be more creative if you have a higher temperature and it will be kind of more like you know follow the the flow chart if you've got a very low temperature and we'll look at some examples in the notebook of exactly how that that looks yeah yeah yeah yeah okay so then you know if I'm thinking about the distribution here if I just wanted to like zoom way in on this and I just just wanted to sort of look at, okay, I've got sort of A, B, and C here. Yeah. You know, what's the sort of way that you think about? So I'm imagining this some sort of normal distribution across each of these, you know, vocabulary choices. And I'm sort of, you know, randomly picking one, but the distribution takes care of the fact that the more probable it is, the more likely I am to pick it. Is that the right way to sort of think about this? Sure. I probably would maybe move away from normal but like uh yes there's a distribution that has a peak and that has a you know a low end that's the beauty of softmax right whatever we have can be fit into that into that zero to one box so it can be expressed as a probability distribution it doesn't mean it has to be normal right with a very high temperature we're going to kind of move more and more towards actually uniform distribution where every token is as likely as the next token. Obviously, your temperature has to be exceedingly high to get to that point in a lot of cases. But that's the idea where we're at some point when we sample from that distribution,'re gonna get a token at random right any of them will do uh and uh you know it's like rolling the dice which token we're gonna select versus with a very low temperature is a chance right where is a chance we're gonna get these kind of like 0.00 to to the minus, you know, tokens, we're probably going to get the one that's 99.9 repeating almost to infinity, right? So it's, this is the idea when we're thinking about that, that softmax that comes after we scale our logits with that temperature value. And this is something else we'll discuss a little bit in the notebook is this idea of we also have another unit we can manipulate, which is that, you know, top K, right? So when we're talking about our tokens, not only can we think about temperature, but we can also say, well, even though we want some variability variability we don't really want it to pick the 18th most likely token or the 15th most likely token right or we only want it to consider the top five percent right right right because we kind of have this intuition that if you consider you know the the the if you have a vocabulary of 128 000 tokens right if you consider the 128 000th choice uh it's probably wrong yeah right the model's pretty confident about that so this idea that we can also kind of we can zoom into a specific set of tokens uh is is incredibly useful yeah like if i'm speaking and I'm not really sure if the next word makes sense, maybe I'll stay away from it. Yeah, that's right. Yeah, you know, but we want some variability in that kind of, you know, in that top 5% of tokens that, you know, are probably right. We want some variability within those, so sure. Yeah, and sometimes you're feeling bold and you're like, I'm not quite sure this fits here, but I'm feeling it right now. That's right. Yeah, I want to go with. Okay, so one more thing I just want to point out here just to make sure I'm thinking about this right. So this is like a token sequence here, this sort of dimension. And so it's important to understand, I think that we've got sort of this sequence of tokens that is in this representation, but then we're predicting only one token. But there is a bunch in the sequence. So the representation actually is of a larger number of tokens than just the one you're predicting, right? And like what is that sort of... How do you think about that? Is that sort of the context window here that we're sort of predicting? What's the sort of easiest way to think about this from a practical perspective? Honestly, we just care about the last one. So I wouldn't think about it too hard. But the idea is we do compute everything and we only really care about the last one, right? So even though we are going to compute a bunch of things, it's kind of like, well, we don't need to predict it, right? Because we already have it. So it's, there's not a ton of use to consider it. So we just consider the last position as the index of the token that we care about. Yeah. And so then as sort of vocab increases it goes from three to ten thousand or a hundred thousand as sort of the uh the number of tokens in the sequence increases as we go from i don't know let's say 8k to 128k then we start to get into the situation where when we do inference we'll really want to take care of those things that don't matter in smart ways right yeah oh yeah yeah yeah well yeah that's a whole another discussion yeah yeah okay good stuff so by zooming in we can also zoom out and this is sort of the the benefit of understanding some of this stuff you said you just said logits i'm going to dig in and we'll come right back to it wiz let's take another look at how we can think about this from an even simpler perspective now. If we zoom in even further, this from the illustrated transformer, and we're going to flip this thing back up the way it was because Jay Almar put this out in a similar fashion years ago. So if we sort of have this decoder stack output here, now I'm not in a sort of token sequence by embedding representation form anymore. I'm in sort of a, I'm just in a single token, a single word in this case, by embedding representation C. And so as I sort of watch this thing move into and through my linear layer, I'm going to get now this number of words in the vocab that I'm going to be able to assess what's the probability of the next word. And of course the way I did this was I came in and I multiplied the attention head weights number of vocab by C here but instead what we have is we have just this 1 by C instead of T by C and so what happens is we end up in a space where we now are going to have a 1 by C times a C by N, giving us a 1 by N. And so that is going to now allow us to take our unnormalized logits and then normalize them, soft max them into these log probabilities that are going to serve as the basis for which I can do simple things like argmax right or I can do more complicated more flexible more interesting things especially if I have a more interesting problem I'm trying to solve in this case maybe I'm just going to select the word straight up. Okay, so this is sort of the one dimension, one word example that you can sort of think about, look at. And I want to kind of bring Chris back up for a little more discussion here before we actually see how to predict this. So we've got now this logits and this logprobs and this softmax. And I just want to kind of break this down for everybody. How do you think about this? What's the difference between logits and logprobs and softmax? Yeah. So logits are in the worst way to say it, right? But the technically correct is the unnormalized, raw unnormalized scores, right? So this is the idea that like, when we project our output, right, from our embedded dimension to our linear layer. So we're moving it into that vocabulary space. We know the attention that's computed is an attention score. That's what we care about. We care about this idea of a score. What is a value that indicates how much we should care about this thing right we quantify that with the with this idea of an attention score when we project that into our vocabulary space what we wind up with is we wind up with a huge set of scores for every token in our vocabulary right and those scores before we do anything to them are called logics. So this is this idea that they're raw and they're unnormalized. So they're just like values, right? And they can be whatever you want. They can be small, they can be negative, they can be positive, they can be large, doesn't matter. But they're scores. When we move to through softmax, right? So when we move through softmax right so when we move through softmax when we take all of those logits and then we use that softmax uh transformation right or this the softmax layer in this case which is just going to apply softmax to those to those scores right that's how we move to those, the probs or the probabilities, right? This is the idea that, you know, probability distributions famously, right, have a few rules that they need to follow. They need to be between zero and one, right? Because they all have to sum to one, the whole distribution has to sum to one. And so this is the idea that we have to sum to one the whole distribution has to sum to one and so this is the idea that we have to transform this kind of mess which is still valuable still incredibly valuable still useful to think about right we can do cool as we just talked with temperature we can do cool stuff with our logits to to help us change what the resultant probability distribution is right but this idea that we have to eventually go to a distribution in order to interact with it in a way that we're used to. So sampling from it, et cetera. And this is that conversion. So the log probes is ultimately what we're going to use to pick a token, right? And we're just going to sample it from that distribution. So that means that- All the jobs, aka Logits softmax. That's right. That's right. Yes. Yeah, absolutely. Yeah. Killer, killer. Okay. So you're going to show us how to disencode? Is that right? We're going to move through it. Yeah. We're going to show everybody. Yeah. Okay. All right. So next up, guys, we've got actually predicting a token. And I'm going to hand this one right off to you, Wiz. Take it from here, man. All righty. Thank you, Greg. So we've got our Colab notebook open. It'll be shared in the chat for you guys. And the idea of this is pretty straightforward. We're going to approach it from kind of the most, the way that we use it, and then we're going to approach it from a from kind of the the most the way that we use it and then we're going to we're going to needle down so first of all we're using the lgpt from carpathy it's just like the best it's nice and minimal it works you'll love to see it we're going to get the additional dependencies that are required for this repository that aren't included in colab the rest of them are just included in colab by default. You will need a GPU environment for this in order for it to work well. We'll clone the repo. We'll move into the repo. And then we'll just look at how it works, right? So we'll say things like we can sample from GB2XL. We're going to put an input sequence of this text. What is the answer to life, the universe, and everything? And then we're going to want a response of one generated sample. And we're going to want a maximum of 100 tokens. And we get like the classic, right? So we get the, what is the answer to life, the universe, and everything? That's what we asked. One possibility is that they have a universe-wide perspective that allows for no contradictions. And if so, then why is there... I mean, this doesn't make sense, but it's related to the question and it's got words like Big Bang Theory. So we're clearly in the right place. This is just GPT-2, right? We've learned to expect a lot more from our language models. But the idea is that we pass in text, and then we receive out text. Now, there's a lot that goes on in between. So let's take a little bit of time to look at it. Number one, you know, we have to get into this. We have to get into this idea of auto regression, right? Causal, right? So these words, what do they mean? And basically what they mean is you take an input and generate a single token and then append that token that was generated back to the input and do it again over and over and over again. Auto regressive, right? So this is the idea. So how does this manifest in code well if we look at the generate function in the nano gpt uh library we're gonna get an insight pretty quick so the generate takes some uh list of indices it takes a max new tokens it takes a temperature and it takes a top k a temperature and it takes a top K. The indices are the actual things we have in our sequence so far, right? So in this case, indices are going to be the tokenized output of this. This bit right here. So what do we do with those? Well, you know, our max new tokens comes in pretty quickly. We just repeat this step, right? A number of times equal to the maximum new tokens, easy peasy. And then we're going to do some truncating. So what we're going to do is make sure that the index fits in our block size. If it doesn't fit in our block size, we're going to do is make sure that the index fits in our block size. If it doesn't fit in our block size, we're going to have a bad time. And the way that we do that is we just lop off the top of it if it doesn't fit in the block size. So we can keep generating forward, but we're going to lose context from the beginning, right? So we talked about this in our long context event we did last week. If you want to go more into the context window. Then we're going to get some logits, right? So classic. We need our logits. Without logits, what are we doing? So the way we do that is we do a forward pass of the model, right? The forward pass of the model is just going to give to us logits and it is going to give to us a potential loss. In this case, loss is going to be none. So we just chuck it into this wildcard, don't care about it. So now we have logits. Remember, the logits are the unnormalized scores. So what this means is that they are just the, they're just a representation, right, of how highly certain tokens scored. And the logits exist across our whole vocabulary. So every token in our vocabulary has a score, right? So this is the idea. We're gonna just not care about all of the other logits in our sequence and we're only gonna get the final set of logics, right? Logics are the next token. So this is the last position in our logics, which you can see here. I know we're kind of dealing with like these 3D arrays, right? So we got these sweet tensors. Thanks, transformers. But the idea is, right, so this is for our batches. So we want to include all of our batches. We only want the last element, right, of our sequence. So that's the last token, what will become the next token. And we want all of the logits for that next token, right? Because that's how we're going to determine what the next token is. So we're then going to scale it by our temperature. We'll look in a second how that math checks out and then of course we're going to use our top k so top k all it's doing is saying you know hey uh we're gonna we're gonna get our top k from our logits and we're gonna either choose our top k or whatever the size of our logits are. This is to say, if we select a top K that is larger than the number of elements in our logits, then we're just going to, that's all of the logits, right? Or that's all of the elements in our logits. So this is all this is doing. And then all we're going to do is we're going to say, hey, you know, for every logit, if it's not being considered, we can't remove it, right? We can't eliminate it. We still need that information because it matters how many elements we have, but we're just going to set them to minus infinity. Minus infinity classically is going to be not selected no matter what transformations we do to it, right? Because it will always be the smallest thing. So it will never be selected. But this is the idea. We basically just say, hey, for every logit, right, that is outside of our top k, boom, you're minus infinity. We don't care about you anymore. Then we do the old softmax arena, right? We do this across the correct dimension to make sure that we are doing softmax for all of those elements in each of the logits. And then we are going to sample from that resultant probability distribution to get our next index. ability distribution to get our next index. And then we are going to concatenate our existing indices with the next index. So what does this mean? Basically, the result of this sampled index is going to be the token that we want next. And we are then going to append it to the next token or to the existing list of tokens. So it's going to become the next token, right? And then we just return at the end, whatever we've whatever we've concatenated. So because we're going to do this in the example, 100 times, we'll have repeated this process of 100 times, we'd have added 100 new tokens, and we're going to return IDXx which is now going to be 100 uh tokens longer and that's that's the thing right so but there's a lot of kind of there's a lot of stuff happening right here and then there's a lot of stuff happening uh kind of kind of here right so let's let's do a little bit of a zoom in so what are logits uh greg's walked us through that so that's's great. And what is temperature doing, right? So I think top K is intuitive, right? We choose the top K logits. So if you are among the highest K logits, you will be kept. Everything else turns into minus infinity, right? So what we're going to do with temperature is we're going to see how temperature influences the resultant probability distribution. So if we have this, let's just pretend our logits are 6, 2, 7, 0.1, minus 8, 9, right? We're making stuff up, doesn't matter, right? If we were using argmax, we'd just say this is the one, right? This index is the one we care about uh so this is what our token is going to be right but of course we are not actually uh going to use argmax right that and you can greedy decoding is great you just always pick the most likely next token every time right there's no chance you do anything else that that is a valid decoding strategy though you might find that the the resultant generations lack creativity and and the like uh so what we're gonna do is we're gonna scale these logits by our temperature in this case is just one and then we're going to do our soft max operation to get a distribution you'll see that we got a pretty wide range of values everywhere from uh you know 8 8.43 to the negative one, all the way down to this, you know, to the negative eight. So very small value. The idea here is that this is the highest, right? So nine is the highest, but the six is, you know, it's not doing so bad, right? We have this seven, right? That's doing a little bit better. So it's not like nine is the only thing that matters in that distribution, right? It's a, if we want to be very crass about it, right? It's like 84% chance we're going to select that nine. But that means that there's a 16% chance that we're not going to select that nine, right? So now if we look at a very low temperature, we can see these logits, they get pretty big, right? So we go from nine all the way to 90. And then when we look at our resultant probability distribution, we can see that all of the others are very small, up to and including one that is to the negative 74, and the 9 is 99.9999998% chance to be selected, right? So when we scale those logits by that low temperature, we make the one that is highest, right, much more likely to be considered. And then the opposite, if we use a large temperature, we can see that they kind of all settle into the same kind of range, right, where we have the instead of 99.99 with low temperature, or about 8085 with with with temperature equal one, we get this, you know, it's only 27% likely to be selected when we use a high temperature, right? Versus the second place, which is 22%. So this is the idea of temperature and how it modifies that resulting softmax operation that we do. And the idea here, again, is, you know, this is a smoother, more close to uniform distribution, which means we're more likely to get, you know, indices that we wouldn't have before. And then, you know, we get a very sharp or distinct distribution. There you go. So, this is the idea. So, for generation, we kind of get this pseudocode, right? For some range, which is user decided, we check and make sure our current range of indices will fit our block size. If we don't, we trim it so it does. We get the logits for the provided indices. We scale the logits. We optionally crop the logits. We apply softmax to converter logits into a probability distribution. We sample from that probability. We add the sampled index back to our input indices. That's the auto-regression coming in, right? And then we're done. So how do we actually get to logits? So we understand we're using logits, right? And those logits exist across that whole vocabulary space, but how do we get to logits, right? We know we're getting attention scores when we're using our our uh our our decoder stacks right we're we have this idea that we're scoring things right where where certain certain uh things are gonna be scored higher based on the relationship or providing surrounding context yada yada so the idea is that we have this idea of an LM head and that all it's doing, right, is it's taking the output of our decoder stacks, which is our input dimension, which is this N embed, right? That's the internal dimension of the model. And then it's going to project it onto our vocabulary size. So this idea that we're going to literally project from our embedding, our internal embedding space or representation, sorry, to our vocabulary. This is that's it. This is a learned process to be very clear, right? It is learned. So this is spoiler alert. This is where loss is going to come in, right? But the idea is that we're just projecting to get to that vocabulary space, right? Because it's not useful. The internal representation of the model is not useful outside of the model, right? So we need some way to project it into a space where it is useful. And we're going to use our linear layer to do that. And that's it. That really that is the that's the thing that does it right. There's there's nothing more complicated in nano gpt in fact there is no softmax that is applied in the model right it's applied afterwards now that doesn't have to be true you can do it in the model if you want but uh the uh nano gpt is going to do this as part of the generation as opposed to part of the actual model so there's no softmax layer in uh in nano gpt now you know what's this like how do we get from logits to loss now right so we've got to tie this into loss somehow so the idea is right we get our decoder block to take input and compute attention scores we project those scores from our internal model dimension onto our vocabulary then we use the obtained raw annular scores logits to find a probability we sample we use the obtained raw, a number of scores, logits, to find a probability. We sample, we append the token, we rinse and repeat. So how does this relate to loss, right? Well, how do we know, how do we know what to score things? And how do we know the scores are correct? Or what's their degree of correctness, right? While we're trading, we need something that we can target, right? This is the whole idea of machine learning. We need labels or targets, right? While we're trading, we need something that we can target, right? This is the whole idea of machine learning. We need labels or targets, right? So how do we know to produce certain scores for certain tokens? This is where loss is going to roll in for us. And with that, I'll pass you guys back to Greg. Yes. Awesome. Masterclass on predicting the next token. And as you guys can probably imagine, we're going to continue this series by covering loss in depth, but we want to give you a little bit of an insight into it. Now, make sure that we're leaving you off at a good stopping point. You know, as was mentioned, in order to train, we have to know loss. And, you know, the sort of logic train is something like this. We want to train models to predict the next most likely token. And so then in order to do that, we need this loss function. But what is loss? Well, if we think about these logits, that's our distribution, let's say, we want to measure the distribution that we have, and we want to make it a little bit closer to the distribution we desire. So we have some training data and that training data represents the next words we would expect to predict. We wanna make sure we're able to do that. So how do we calculate this loss exactly? Well, interestingly, and unsurprisingly, if you've got a machine learning background, perhaps, we use cross entropy loss. Now, this is something you might be familiar with. This is going to be the topic of our next event on this, where we dive into lots of depth on cross entropy and exactly what's going on but this does connect back to our initial hot take that lms are kind of classifiers okay when we talk about cross entropy here the question is what are we looking at the cross entropy of between which two things are we looking, and we're looking at the logits and also the targets. And the way that we sort of differentiate the logits and the targets is we can think about the logits as being the sequence up to the next word. And we can think about the targets as being a shifted sequence up to the word we want to predict. In order to train, we need input, we need output. And interestingly, this is what it looks like down in the nano GPT code for the cross entropy calculation. If we don't want to train, that means if targets is not available for us because we're not training, what are we doing? We're doing inference. And when we do inference, we're using that language model head on the very last piece. on the very last piece. So here we can see this sort of auto-regressive nature where we're predicting future values based on past values of our decoder stack transformer. We're shifting as we go during training. And this is what we will give you some insight into in our next event. I want to go ahead and bring Wiz back up to the stage to discuss a little bit about our question we began with. And, you know, we can think about this difference between logits and targets as manifesting in this shift, Wiz, but maybe we can just end on this question and come back uh to exactly where we are next time and you guys shout out in the chat any questions that you have or please use slido but are transformers just classifiers wiz uh what's going on here have we have we seen that they kind of are yeah i mean kind of is exactly right they're kind of class. I mean, kind of is exactly right. They're kind of classifiers. I mean, they do classify right, like, we're not going to use them like classifiers very, very, you know, specifically, and if we wanted to, we'd use a different head. But the idea is, yeah, I mean, they're classifying in some sense, definitely, right right we are predicting of a possible set of uh classes or in this case else com tokens right which is the which is the most likely uh you know of of the batch now the way we're using the model and the auto regression kind of moves into prediction task right but uh the guts of it are very close to uh you know straightforward classification so we're using auto regression and classification then is that is that what we're doing yeah i mean uh it somehow it's not surprising right it's not no we have to pick the next token which means we have to know what is the, given what we have so far, what, what token follows, right? Like it's a, uh, we, we have to pick, uh, of, of a set of tokens, the next one. And so there's, there's fields a little classify even without knowing the internals. Yeah, yeah, yeah, yeah. Okay, well, I wanna go ahead and invite everybody to add your questions and we're gonna go try to cover as many as we can, but question from the audience was, how do we actually train these networks when there are multiple possible answers? Say that again? How do we actually train these networks when there are multiple possible answers? Oh, that's why we that's why we need that probability distribution. That's why we want sample it, right? So the idea is there could be more than one token that essentially works, right? So what if we have two tokens that have, you know, an equivalent, you know, ish probability, they're never gonna be exactly, but like equivalent ish probability, this is the benefit of sampling. Sometimes we'll choose one, sometimes we'll choose the other, right? So it's like synonyms in English or whatever. You know, you, you don't, you can choose whichever of the, of the words you wish to, as long as it makes sense. And using the sampling strategy and the temperature strategy helps us to better emulate that. I mean, obviously, you might be leaning more towards one than the other, depending on like the who you're talking to, or perhaps what you've said previously, right to keep things consistent. But this is the idea that when we're training, and we're using this, this logits softmax approach we actually can capture this behavior where you know two tokens are equally likely or five tokens are equally likely equally in air quotes okay yeah so another training question here people are excited for the next training event i can see wouldn't it be better to train the neural network only up to logits with argmax without using softmax part during the training? Yeah, we don't use softmax part during the training. We just use the logits. Just the logits without argmax? Just the logits. Just the logits. Okay. Because the whole thing we just discussed, right? Yeah, that's right. The distribution and- We give them a pile of labels and a pile of produced logits, and we go from there. Yeah. This last question, how do we incorporate context to make our choice on the next word, I think is what this question is asking. Yes. This is sort of a raggy question, I think. So, I mean, they're the same token, right? Our logits will be high for that token in, let's say to simplify things a lot right we're going to say that the score for that token is going to be high for either context right uh because it can fit in either context so uh you know both when we're talking about you know weight and when we're talking about cash you know uh especially we're talking about Britain in in cash right we're talking about cash, you know, especially we're talking about Britain and cash, right? We're going to see pound to score highly as a potential next token. So this is the idea, right? Like the scores per token are generated based on all that context. So that's why we need to pass in some amount of text first in order to get a score that is good for that for that next token this is why you know things like few shot uh you know many shot work in context learning works right because essentially what it's doing is it's modifying the scores of the potential next token by all this stuff that came beforehand right got it got it okay well so next up was then what what uh so what are we going to cover we're going to cover sort of like dive down deep into cross entropy get a feel for what loss is try to understand exactly how useful this is in training what it looks like and from there we're going to be able to really open up exactly everything that's kind of at the center of a lot of these more complex techniques aren't we there's a lot going on with the loss functions yes we're we're we're finally at a place where we can start talking about loss, and it makes some sense to be able to be tied back to what we actually are going to do. All right. All right. So I'm going to go ahead and wrap up. In conclusion, everybody, thank you for joining us today. We saw that the decoder stack GPT provides us with an output that is in embedding space. We then want to put that output from embedding space into natural language space. We can do that through this calculation of the logits that is going to take into account our vocabulary information from that language model head, and we can eventually choose softly, perhaps, what the next best word is. When we look at inference versus training, we can look at logits that help us do inference, and we can look at what the targets are as we slide that window to try to understand exactly what's going on at the root of training. We're going to dig into that next time as we continue our journey into loss functions and deep down into the transformer, through which we'll see a lot of interesting things, including how to optimize running all of this stuff in a compute-efficient way on your hardware and much, much more. So if you enjoyed this event definitely like and subscribe on youtube that is a wrap for our session today and if you would like to follow along with the discussion that we have actively going on in our community all the time and folks in the chat definitely know this and are part of it then please join our discord community we'd love to have you we've got an event coming up in just a few minutes here with one of our best community members. And we are weekly telling stories about some of the most successful folks. I believe we got shouted out in the chat today, Garrett, and what they're out there building, shipping, and sharing. We just kicked off our most recent version of our AI engineering bootcamp last night. it was a banger. And if you're interested in jumping into the next cohort, please reach out to me and let me know if you have any questions. I'm happy to answer for you. And finally, if you have any other feedback at all, we'd love to hear it. If you have any content you'd love to see, we'd love to know about it. And until next time, guys, we'll keep building, shipping, and sharing, and we hope you do the same. We'll see you soon. Bye, everybody. Have a great rest of your week.
The Next Token: How LLMs Predict
Join in to learn about the foundational aspects of prompt engineering, retrieval augmented generation, fine-tuning, and agents, to exploring the technical nuances of LLM operations like prompt tuning and the intricacies of token prediction, this event is your gateway to mastering LLM application building. Discover how to effectively manage applications in production, optimize performance, and intelligently evaluate outputs. Whether you're an AI Engineer, a leader in the field, or simply keen on the latest AI technologies, this session promises a comprehensive breakdown of both the theoretical and practical aspects of modern LLMs. Don't miss the chance to expand your understanding from the fundamental mechanisms of token prediction to the advanced strategies of unsupervised pretraining and beyond. Click now to join our "Everything is Loss" series and start unraveling the complex yet fascinating world of LLMs!

Event page: https://lu.ma/nextoken

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/1FixiyoBRqad346PixnFcc

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

For team leaders, check out!
https://aimakerspace.io/gen-ai-upskilling-for-teams/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/WmVKUE3bfAoi1KDS8