 Hey Chris, is it true that we can improve on our PEFT-LORA approach with this quantization thing? It sure is, Greg. Yes. And is quantization like really as good and as dope as everybody's talking about? Yes. Emphatically, yes. Emphatically, yes. Man, I cannot wait to see exactly what's going on inside. You're going to show us how to do this today, right? Sure. All right. Let's go ahead and get right into it, man. We'll see you back in just a little bit. Today, we're going to talk quantization. I'm Greg. That's Chris. We're from AI Makerspace. This is a bit of an add on to last week's event, which talked about parameter efficient fine tuning and low rank adaptation. Today, we're gonna take it to the next level and talk quantization. We'll demystify the idea of quantization, and we will also talk about how to leverage the latest in low ink adaptation which is a quantized version of it called QLORA as always we'll be collecting questions with slido so go ahead and provide your questions for us throughout the day at that link and then we'll go ahead and answer as many as we can when we're through with the demo at the end. Of course, we'll have Chris back to lead and wizard his way through the demo on quantization soon, but for now, let's cover what we need to know so that's going to make sense to us. We're going to talk quantization of LLMs today, and we're going to talk fine-tuning with LoRa. This is the main goal. We want to understand and we want to align our aim to really grokking QLoRa and then seeing how we can implement that. We got a little bit of insight into quantization last time when we were loading the model but now we want to take a look at how it can be used to fine tune and some of the background and intuition associated with why this works and what the industry has sort of learned about the precision of numbers within our llms so we're going to talk-tuning quantization QLORA, and then we'll do it. And to sort of contextualize this, similar to last time, we wanna understand that often fine-tuning is coming into play after we do prompt engineering, often after we set up a retrieval augmented generation system. And we wanna now take a look at how we can optimize our large language model, or in other words, how we want the model to act, how we want the input and output schema of the model to be a little bit more constrained, a little bit more dialed in, a little bit less large, a little bit more small. And this is sort of the trend we're noticing as 2024 is upon us now. We are seeing a bigger and bigger interest in smaller, more performant language models and fine tuning is really a key aspect that's going to help us to get there so let's just remind ourselves what we talk about when we talk about fine tuning with peft laura PEFT LoRa. And why we need to do this. You know, when we talk LLMs, they're super big. They have billions and tens of billions of parameters. It's likely we'll see models with hundreds of billions of parameters before too long. Not all models are always getting bigger, but some of them are. And the reason is, is because if we keep adding more text and more parameters, we are pretty confident that our next word prediction will continue to improve. prediction will continue to improve. But as we do this, as we build larger and larger models, as we have to deal with more and more compute in order to be able to handle them, whether that's loading them, training them, fine tuning them, or performing inference on them and serving them. We're kind of abstracting away from the regular developer, the regular individual out there that doesn't have access to a giant cluster of GPUs to be able to even play with these things. And this is the core problem, is that when we go and we want to do full fine-tuning on many, many billions of parameters, this becomes a huge pain for anybody trying to use consumer hardware, any small business trying to just use the laptops that they have, maybe a few resources on the cloud. And this is as true for fine tuning as it is for loading and storing, certainly for deploying these models. It just costs too much. And the solution for kind of dealing with the fine tuning, the storing and the deploying is kind of the same. But today we're focusing on fine tuning. Today we're focusing on fine tuning using fewer parameters. It's all about using fewer parameters. We don't need all of them as we started to get some intuition into last time. And in fact, the ones that we have, what we're going to do today is we're going to take those parameters and we're going to make them smaller in a sense. We're going to make them smaller in a computational sense. This is the essence of quantization. So while it may not be necessarily fewer parameters when we talk about quantization, although it often is when we talk about fine-tuning, we're just trying to move these big, big, big models towards smaller packages through fewer parameters and through more efficient representation of those parameters. And we saw last time, we saw that LoRa is the number one PEF method you should know. It's called low-rank adaptation. And the big idea of LoRa, as we discussed, was to fine-tune using factorized matrices. And again, we didn't focus on fine-tuning absolutely everything. We did fewer parameters. That was great because it was more efficient. And we found out that we could actually leverage LoRa adapters for many tasks. So you could have one big, big model and a ton of different lower adapters and deploy that to production. Deploy each of those adapters to production because at inference is when the adapter would actually come into play. So very, very flexible, very good technique for. Larger companies and industry, especially that want to just have many adapters and larger companies and industry, especially that want to just have many adapters in one very powerful model, we'll probably start to see this emerge as an approach to AI development in the enterprise. And, you know, it's really comparable to fine tuning, full fine tuning. full fine-tuning. So, you know, we saw, in essence, that fine-tuning is all about modifying behavior of LLMs to update parameters. Parameter-efficient fine-tuning is all about fine-tuning with fewer parameters. Low-rank adaptation was all about fine-tuning using factorized matrices. And so parameter-efficient fine-tuning through low-rank adaptation is all about modifying behavior by updating fewer parameters using factorized matrices. So this all sort of flows together. This leads us directly to our new friend, quantization. And this meme is so good, I had to put it twice, because it's such an oft misunderstood idea. Certainly has taken a long time for me personally to really try to grok this thing. So let's see if we can break it down in a way that makes sense to all of you. First off, the weights of our LLM, when we talk about weights, it's the same thing as when we talk about parameters. Okay. So parameters, I might say weights, we're still talking about parameters. Those parameters are simply numbers. They're just numbers. And specifically, they're floating point numbers, They're floating point numbers, also known as floats. And it's important to understand a little bit of the detail here, because this is the essence of what we're doing in quantization. When we talk about floats, you may harken back to your days in school, maybe chemistry, back to your days in school, maybe chemistry, where you learned about significant figures, sig figs, everybody's favorite, right? And then if you're like me, you become an engineer and you don't care anymore, ever again. But I was a mechanical engineer. If you're a computer scientist, computer engineer, maybe you continue to go deeper. And these days in AI, if you're a developer, you need to continue to go a little deeper. Because this idea of a float is cool, this integer with a fixed precision, we can talk about representing, for instance, 12.345 as 1, 2, 3, 4, 5 times 10 to the minus 3. And we can then do this by using a specific number of bits in our computer. When we talk about this precision, this fixed precision, there's a number of different types of precision. What we're going to generally be using is what's called full precision when we're doing computations that are kind of default computations. Full precision means that I have 32 bits to represent my floating point number. And they're broken up into a couple different pieces here, but the big idea is that there's 32 bits. And the question is, is that the right amount when we want to go and deal with 70 billion parameter models and things like that? And it turns out that in machine learning, we found sort of over time through experiments that if we didn't use 32-bit precision and instead we used 16-bit precision, Instead, we used 16-bit precision, essentially half precision, to again, simply represent those decimal numbers that are inside of each of the neural network, that represent each of the neural network weights, sort of each of the neural network perceptrons is a way you could think about this. Then what we're finding is that we can get almost identical inference outcomes from our LLM. Because remember, we just want the words that come out at the end. We just want the ideas that come out of that. We just want the outputs. We don't necessarily care about the precision of the stuff within the black box. We put in, we get out. And a lot of people were seeing this. A lot of researchers were seeing this with the large language models, that if we just leveraged half precision we can get very very good outcomes and what this does is this effectively halves the entire model size so what are we saying we're saying that we can sort of get exactly the same thing coming out coming out, even if we represent each of the model weights using half as much information we can think about. Because really, I mean, how many sig figs do we need? And another way we can talk about moving from a 32-bit down to a 16-bit representation is we can say we are quantizing. We quantize the 32-bit weights down to 16-bit. weights down to 16 bit. Hence quantization. Now, when it comes to quantization, there are many different approaches to quantize model weights. So, this is very important. We're not going to cover every single approach because that's not really necessary for what we want to discuss today. But there are many different ways to quantize model weights, and we hope to continue to bring you more content on ways that are a little bit different in terms of their actual implementation and the nuances in future content but for today we're going to focus and use this Q-Laura idea as a focusing lens now Q-Laura starts the story begins with a paper called 8-Bit Optimizers via Blockwise Quantization. And this was a paper that came out of the University of Washington. Tim Detmers was the lead author, and he's been quite a superstar in the field.'s he's kind of like the quantization guy and in this paper they showed that you can use 8-bit representations and maintain performance that we're seeing at a level of full precision or 32-bit. So here we see in this kind of early paper, again, one of these pieces of work where they're saying, hey, look, experimentally, we're seeing that if we reduce the precision, we can still get great results. And this is not reducing it to half precision it's reducing it to quarter precision 32 down to eight and this bits and bytes approach this bits and bytes paper turned into what became the bits and bytes library which has since evolved and is something that we'll see the Bits and Bytes library, which has since evolved and is something that we'll see Chris use today, and it's something that gets used all the time now. Now, Bits and Bytes, you can go ahead and recall that one byte is equal to eight bits. We're going to continue the discussion in bits today, but you'll see many papers and discussions of things that will talk in bytes as well. So pretty simple to understand why the library was named bits and bytes. Now, again, this is one approach. And so there are some trade-offs as there are with any approach. For instance, when we use the bits and bytes approach to quantization, we're not really getting any additional benefits to our inference latency. We're not really speeding up inference a whole lot by using this particular approach to quantization. However, what we are doing is we're leveraging a tool that gives us very flexible use of those LoRa adapters, right? So for enterprise, if we're thinking about how do I have one big model and just have a bunch of adapters, this is going to be our friend. And this is why we choose to focus on this one today. And this bits and bytes library forms the basis for what comes next. It kind of forms the basis for this QLORA idea, this efficient fine-tuning using quantization. And the fine-tuning using quantization from the QLORA paper, the big bang box takeaway of this is it's super great, even though it's eight times less precise. less precise. So what we actually have going on in QLORA is we have not an 8-bit representation, but we have a 4-bit representation. And so what completely insane. And we can fit all of that on a single 48 gig GPU, single 48 gig gpu which is like just kind of incredible it's just kind of it's kind of mind-blowing that we can do this and so this q laura paper is essentially coming and saying hey hey, listen, we've got this idea that we can do fine-tuning using a four-bit approach versus even a half-precision approach, and we get amazing results. And so this is the essence of what's going on here with QLORA. the essence of what's going on here with QLORA. And so what we can kind of think about is if we go back to this idea of PEPF-DLORA fine-tuning, where we're modifying behavior by updating fewer parameters using factorized matrices. And we add this idea of quantization, where quantization is simply representing high precision numbers with low precision. Then we get to this place where we talk about PEFT-QLORA fine-tuning, where we talk about PEFT QLORA fine-tuning, where we're modifying behavior by updating fewer quantized parameters using factorized matrices. And so the process as outlined in the QLORA paper and the process that you're going to see today is something like this. We download the model weights. Anytime you download model weights from Hugging Face, they're always going to be in full precision, 32-bit. Then we load our parameter efficient fine-tuning model into GPU memory. Anytime we load into GPU memory for inference or training, we're going to be loading using that parameter efficient fine tuning method. And then we'll initialize our low rank adaptation, our LoRa configuration. And finally, and this is the key, this is the key to the whole thing, is that during training, what happens is we have the full precision 32-bit model, and we're going to actually load the 4-bit model, quantize 32-bit down to 4-bit, for training. Quantize 32-bit down to 4-bit for training. Now, during training, we're going to flow through the network, and we're going to, as necessary, each time we have to do a computation, each time we have to calculate something during our training process, we're going to de-quantize that 4-bit representation back up to a 16-bit half-precision representation. We're going to do the calculation, and then we're going to re-quantize back down. And at each step of our training or fine-tuning, we're going to quantize, de-quantize, move on. So we're never holding that half precision fully in our GPU memory. But rather, we're simply using half precision to do the calculations. This is the magic of what's really going on behind the scenes. And it turns out this works incredibly well. And again, that intuition behind the 16-bit piece is that we saw that for inference, you can go from 32- down to 16 bit and get very very good results we saw this experimentally over a lot of time not just papers from the university of washington but also papers from many other researchers and this q laura approach fundamentally Fundamentally, is to load those full precision weights into GPU memory as quantized 4-bit weights. And then only de-quantize up to 16-bit during calculation. Back down as it moves through. All right. So this is the core approach that we're going to see today. You're going to see things like this. This is the bits and bytes configuration. And you'll notice when we want to load in, we want to load in in 4-bit. You're also going to see a data type called NF4. Chris is going to talk a little bit more about it. It's very important. It's very essential to the QLOR approach. And that's it for the big ideas we need to really see how this build can be taken to the next level. So what we wanna do is we wanna take the same build that we've already looked at, the old UNO reverse card build, given the response, predict the instruction. We want to use the same model that we saw last week because it's still one of the best out there. Mistral 7B instruct V0.2. And we're going to use the same data for fine tuning. Just keep everything simple. That Alpaca GPT-4 data set is there. So again, output response, predict input instruction. And with that, we're ready to kick it back over to Chris, the wizard, to show us how to do fine tuning with PATHQ, Laura, and fill in some additional details. Wiz, over to you, man. Q Laura and fill in some additional details. Wiz over to you, man. Oh yeah, thanks Greg. Really appreciate it. And guys, I'm excited because quantization is definitely one of my favorite topics. It is the kind of like one of the best things we could do right now. And as you can see, we only used around 20 gigabytes of GPU RAM to train this 7 billion parameter model, which is quite impressive in my lens. That includes fine tuning. In any case, we'll get right into it. First of all, we're going to be using Mistral 7B Instruct V02. This is just Mistral's most recent Instruct tune model. I love it. And we're going to now move on from PEFT, which we discussed last week, into the Q in QLORA. So we discovered or we discussed, you know, the idea of how we can reduce the number of parameters that we train. But now how do we reduce the size of the parameters that we train? Now, first of all, what is quantization? Greg already talked us through it. I'm going to give a brief overview here of what's happening under the hood, and then we'll get into how to implement it in code. Spoiler alert, it's super easy. Thanks, bits and bytes. But let's look at what quantization is from this perspective. So quantization is a process of discretizing an input from a representation that holds more information to represent a representation with less information right that's crazy so the idea is we want to express more information with less information so how do we actually do that well in the tim detmer's q laura paper they rely on this process called blockwise k-bit quantization which which sounds, you know, like, very, you know, scary, but it's not so bad. It relies on two very important things. One, it relies on the fact that in neural networks, the model weights are mostly normally distributed. So as soon as we, if you're, if you're coming from a stats background, as soon as you hear that word normal distribution you you know your your eyes should light up uh you know we're we're going to be able to make use of a lot of very clever tricks uh to help us do whatever we're trying to do um and then it also relies on this idea of the nf4 format which which is a number format or data type created by Tim Detmers and team, which is information theoretically optimal. Now, not literally, it was proven this is not literally true, but it is empirically, for all intents and purposes, this is a fact that NF4 is very, very efficient, which is excellent. So how does this work behind the scenes, right? So, okay, we get it. Model weights are normally distributed. That's great. So what we're going to do is we're going to essentially put a pin in the number line that is near to the mean, right, of our desired numbers, which are going to be in a distribution. And that distribution is going to be normal, right? And then we're going to kind of use that mean as a zero point. And we're going to use this NF4 data type, which is a zero centered number format to represent the numbers that appear around that specific point in the number line. So there's a step that needs to take place here. We're going to normalize all of our numbers to be within a specific range of minus one to one. And then we're going to be able to have this idea of a saved place on our number line that we're going to understand a range around. And that's really about it. Now, it's a bit simplified and it's definitely, you know, you can look at the paper for the math. It's great. But the idea is that we have, we kind of drop a pin in the number line and we have this NF4 number format, which represents a range around that point to the number line. And that is what's going to build up the buckets or bins that we're going to use to represent our numbers. And the reason this works so well is again, because of the fact that model weights are normally distributed and because this is an informationally, theoretically optimal data type for that minus one to one range. So this is specific Yennefors for that minus one to one range for normally distributed, to one range. So this is specific, the n of four is for that minus one to one range for normally distributed, well, distribution. So that means the only reason this works is because of this first fact, right? Now, beyond just that, QLORA does an extra step. So you might have thought to yourself when I said drop a pin in the number line, right? Well, okay, if we drop a pin in the number line, that's all well and good, but doesn't that mean that we have kind of like a high precision number, right? It doesn't have to be as high precision perhaps, but it's definitely still high precision. And that's true, right? That pin we drop is high precision. Well, it can be used to represent many numbers. In this case, you know, 64 numbers from the QLORA paper. So each pin is associated with 64 numbers. Tim Demers and crew said that's not enough. You know, that's going to give us 0.5 bits per parameter of overhead, right? So we need to go bigger. So what they did is they actually took all of those quantization constants. That's the technical term for that pin that we're dropping, right? We take those quantization constants, and then we also quantize those. So we represent our quantization constants in an 8-bit format, and we do 256 of those for every 32-bit precision number. So we have one 32-bit precision quantization constant that sits on top of 256 8-bit quantization constants, which sits on top of each of those sits on top of 256 8-bit quantization constants, which sits on top of, each of those sits on top of 64 4-bit. So you can see the savings in terms of memory here is insane, right? We're able to represent so much of our data in that 4-bit representation. And we're also able to do it in a way that retains a ton of information. And that is key. I saw some questions in the YouTube chat kind of concerning, you know, what's the trade-offs here? What's the performance gains? And there definitely is some when it comes to latency. We'll discuss those as we move through the rest of the notebook. But in terms of the actual effectiveness of the model, the performance hit can be very small. It is not zero. There is a performance hit, but it's incredibly small, which makes this a very effective technique, especially when applied in the way we're going to see it applied today. So that's basically what we're talking about when we talk about this idea of QLora, right? We're talking about dropping a pin on the number line and then saving kind of numbers or representing numbers around that and then doing that one more step abstracted which is harder to visualize but there it is okay so how do we do it in code now right uh well first of all we gotta load our our kind of familiar usual suspects here so we're bits and bytes data sets accelerate uh the laura lib library transformers and peft uh these are all kind of staple libraries we're bits and bytes data sets accelerate the Laura lib library Transformers and peft these are all kind of staple libraries we're going to be using when we're using these uh kind of Q Laura tools and then we're going to grab our model and the model we're going to grab is the Mistral AI Mistral 7B instruct v 0.2 it's the most recent uh instruct model for Mistral it's a great one and then this is kind of uh you know where the magic happens this is the bits and bytes config uh this is from the bits and bytes library we're gonna see that we load in four bit so this means when we actually move our model from those saved weights uh that exist on our on our drive, when we load those into our GPU, we're going to load them in that four-bit quantized state, right? So that's that collection of numbers and then their quantization constants and then their quantization constants because we're using this use double quant, right? If we omitted that use double quant, we would only do one step, and then we would be saving less effective memory. We're also going to be using the quant type of that NF4 I talked about. That's the Tim Detmers and crew created number type, which is information theoretically optimal. Again, not literally true, but it's close enough, so we'll keep saying it. And then we're going to have this idea of a compute D type, which is going to be torch B float 16. Now this is very important, right? So when we store numbers in 4-bit, that's awesome. But when we try to compute with them, it's really bad. It's actually quite bad, right? If you think about when you multiply two numbers together, especially if they're kind of small, right? If you think about when you multiply two numbers together, especially if they're kind of small, right? We usually wind up with a number that is relatively needs more precision to fully accurately understand it, right? When we divide 100 by 1000, we wind up with a very, you know, a small number. And the idea is that we'll need more precision to represent that very small number. So what we do with the QLORA approach is we actually de-quantize whatever we need to compute with our weights. Now, this is done at a per-tensor level. So we never have the full model de quantized in memory, just one tensor at a time, right? So this saves us a ton of a ton of space. And it also lets us have the ability of computing as if we have this model in that higher precision or B float 16 format, right? Which is huge. So we're saving tons of space and then we're de-quantizing. So we also retain some of that compute precision. And that is what lets this method really shine, right? The fact that we de-quantize for computation and then we store in 4-bit. I think without that, this would be a less powerful method. But with that, it's amazing. You can choose up to full precision here. Obviously, that is going to come with some small memory overhead. You do have to upcast a tensor to the full precision, but it's negligible compared to the size of the model. And it does also, and this is critical, it does come with some inference and training latency overhead, right? The fact that we have to de-quantize and re-quantize, de-quantize and re-quantize, this means that we're performing an additional operation per computation. And so that is going to impact inference. Now, Tim and team have written some great kernels for this. So it's not very slow, but it is going to be slower than if we weren't doing that extra operation. And so this is one of the key trade-offs, right? We had questions about trade-offs. One of the key trade tradeoffs with Qlora and with the bits and bytes approach is that it is extraordinarily flexible. It is very powerful and it works very well with a PEFT adapter methods. So like LoRa and others, but it does cost us a little bit of inference latency in training time. So that's important to keep in mind. Once we have our bits and bytes config loaded, all we have to do now is just load our model like we normally would. So auto model for causal LM from pre-trained. We're gonna pass in our mistral AI model. We're gonna pass in our quantization config. We're not gonna need the cache and we're gonna map this to auto, which is gonna shove as much as it can into our GPU. In this case, again, because the actual model loaded only takes up about 15 gigabytes of GPU memory, it's all squeezed into the GPU there. So that's great. We do some pre-processing on our tokenizer to make sure that it's set up in the right format for training. And then we can look at our model architecture. You'll notice that we have this four-bit layer, right? This four-bit layer is where that bits and bytes comes in. You'll see that we have the four-bit layer on our QKVO proj as well as our MLP. So it's all four bit, all the way down. This is the idea, right? We don't want to just quantize some of the model. We're gonna quantize as much of it as we can. However, you will notice that we omit some of the layers, specifically we omit our layer norms. And the reason we omit our layer norms is we know that our layer norms. And the reason we omit our layer norms is we know that our layer norms are going to tend to a very, very small number, you know, near zero. And we're going to run into some training instability issues if we use lower precision to represent these layers. So we're actually going to keep those in full precision. Now they're very small compared to their weight matrix counterparts, but we do want to make sure that we're keeping those layer norms in a higher precision. This is to avoid training instability issues, right? If we have these numbers kind of diverge and cause a ruckus, right? We're not going to be able to train very well. And so that's why we don't see those four-bit layers here. Now that we have our model loaded, we can see that it's in four-bit. We're very happy about that. It's time to peftify it. We talked about peft last week, so we're not going to spend too much time on it today, but the idea is fairly straightforward. We are going to use our LoRa config to set up our rank. Our rank is going to be 64 in this case. We're going to use our LoRa config to set up our rank. Our rank is going to be 64 in this case. We're going to set our alpha, which should be by conventional wisdom, about twice your rank. Though you're, you know, again, it's always worth doing hyperparameter searches here to make sure you have the most optimal hyperparameters. Your LoRa dropout, pretty consistent value. Your bias is none. Task type is causal, because that's what we're doing. You'll also notice that we have our QVK proj modules. We, again, with QLoRa, we want to target as many modules as we can, right? The QLoRa paper's wisdom is that we should actually target all possible layers of LoRa. In this case, we're just going to leave it up to PEFT to simplify things a bit for us. For our base model, all we have to do is prepare our model for k-bit training. This makes sure that we can train and that all of the trainable layers are set appropriately and that any frozen layers are also set appropriately. And then we're going to get our PEFT model and our PEFT model is going to uh give us those laura layers now you'll notice that we have only 2.7 million trainable parameters out of a possible many billion trainable parameters right and the key thing about q the q and q laura right is well is great, when we make each of these parameters one eighth the size, right, we're effectively reducing this by another factor of about eight. It's not strictly eight because of the fact that it doesn't interact with all layers, but the idea is it's about eight another factor of eight reduction in the uh in the total size of parameters that we have to train which is insane right it's uh we we went from kind of we're already at a fraction of a percentage and then we even further reduce uh the amount of actual uh work that we have to do, which is great. And then we can see here that our LoRa layers are also 4-bit, right? We have our LoRa layers are 4-bit as well as our actual, you know, regular layers that were converted to 4-bit. After that, we're going to load some data. We're just going to grab the Apaka GPT-4 data. We're going to do this Uno reverse card train, just a fun one. It's kind of like the classic now. I think this is what you're going to see. Whenever you do an instruction tune, it's just fun and it really proves the point that the process works. So we're going to ask the model to take a input and then generate an instruction. So we're going to create a model that's good at generating instructions. We're going to use this generate prompt helper function in order to create these prompts that our model will be trained on. And then we're going to set up our trainer. Our trainer, this is all boilerplate. The other big insight from the QLora paper is this paged Atom W32 bit optimizer. I'm not going to go too into it here, but the idea is that this idea of using paged memory is really, really effective, and it helps us train very stably and very efficiently with very little cost to us other than we have to flag it. The rest of this is all boilerplate, right? It's good boilerplate, but it is boilerplate. And we are going to make sure that we have our BF16 equals true, which is going to make sure that our compute D type is compatible when we upcast, which is necessary. It says CUDA, but would a Mac suffice to fine tune the model to the 4-bit? I would recommend a GPU, a NVIDIA GPU for sure. The kernels are written for it. I believe you can use 4-bit on other devices, but it's not necessarily going to be as efficient or as fast. The optimization of the kernel really added some speed to this process but i'll get back to you uh more about that uh after a little bit of digging to make sure that you can do this on mac even if it is going to be slightly less efficient uh we're going to use the trl trainer the sft trainer from trl in order to train our, our max sequence length of 2048 just for Mistral itself, and then we can train this using trainer.train. At the end of the day, we reload our model, just a quirk of path. We reload it. We make sure we load it in 4-bit, and then we have our torch D type for float 16. That's the compute D type again. And then we are going to, you know, look at the model. So we say in instruction, identify the odd one out among Twitter, Instagram, and Telegram. That's great. That is, that's an instruction that would result in this, in this, you know, in this kind of the odd one out is Telegram response. And you can see the ground truth is identify the odd one out is telegram response. And you can see the ground truth is identify the odd one out. And if we look at the base model, we can see that the base model's instruction is much less good. It does not even mention telegram. And so, not a very good instruction. But that is it for me and the code demo. So with that, I will pass you back to Greg who will wrap us up. So with that, I will pass you back to Greg. We'll wrap us up. Yeah, thanks, Chris. That was awesome as usual and love that deep dive explanation on exactly what's happening within the quantization method in the QLORA paper. So today we saw building off this PEFT-LORA approach, Today, we saw building off this PEFT-LORA approach, that PEFT-qLORA fine tuning is really about modifying behavior by updating fewer quantized parameters using factorized matrices. So this idea of using fewer parameters and of using the LoRa factorized matrix approach. This gets us from 3.8 billion down to 2.7 million parameters, less than 1%. And then we come in with quantization. This is technically blockwise k-bit quantization, effectively just allowing us to express more information with less. And the key to the QLoRa method is that from that 2.7 million parameter level we're coming in and we're starting to actually quantize that down to four bit before we we begin training during training we will de-quantize when we have to do computations and before re-quantizing to continue the training process. Next week, we are going to be covering how to not fine-tuning and loading, but now serving an inference with VLLM. So we hope you can join us for that one. But for today, we're going to go ahead and get started with the Q&A period. I'd love to invite Chris back up to the stage. And if you guys have questions, it looks like Manny is crushing it in the Slido right now. So shout out to Manny as usual. But if you guys have questions, crushing it in the Slido right now. So shout out to Manny as usual. But if you guys have questions, throw it in the Slido. We'll also try to get to your questions if you throw them in the YouTube live chat. But Chris, let's go ahead and jump right into it here. First question. Is the reason we don't get inference latency benefit with QLORA because model weights are re model weights are retained as 32 bit during inference. I mean, I, yeah, I mean the question, uh, to be more specific about, uh, the phrasing, I think we could say that the, the model weights are de-quantized to a higher precision during inference. So yes, that is why we don't see a benefit to inference. In fact, we see a penalty. It's not a big penalty, but there is a penalty. And so, but yes, that's exactly why. Oh, okay. Nice, nice. Yeah, excellent question. Astute one there. And then first one from Manny here. When we're talking about parameters, are we referring to additional features such as Xs in the equation, Y equals predict X1, X2, Xn? Are X1 to Xn considered parameters? What are we talking about when we say parameters? Yeah, parameters, features, it's all numbers, weights. I mean, we have so many different names for similar kinds of objects. I would think of parameters more specifically as the entities that fill up these weight matrices that we use to compute when we're actually doing that matrix multiplication. But yes, I mean, essentially a parameter, a parameter is any node in the, in the model architecture, right? So this is not something that you're going to want to use with like your XG boosts or your, you know, your kind of traditional ML methods, right? It's not like a random floor forest applicable, you know, technique. It's specific to that deep neural architecture. And it's also specific right now to that transformer architecture, though there's no reason it needs to be. It is most explored in that space. Hopefully that answers the question, Manny. Yeah, yeah. Well, we'll kind of flow through some of these other questions and pop back to Manny's questions as well. I think this one's super relevant to everybody. If I don't have a powerful laptop, where can I practice these techniques? Honey, P, it's Colab. Get yourself into Colab. Colab makes it so easy. And the whole benefit of this kind of thing is we can load these very large models with very little resource. And so oftentimes, you can load like a 3 billion or 6 billion parameter model, you can load that in a free instance of Colab right using the free free tier GPU, the T four. So it's I think that's a great way to start if you don't have a powerful laptop. As you get more embroiled in the space, you might look at other cloud hosting solutions, Lambda or AWS, whatever you want. But for the getting started beginner, I would say Colab is your best friend. If you want to, you can pay for compute so you can pay to get a little bit more uh beefy gpus but uh stick to the free tier and and stick with your kind of three to six billion parameter models and you're gonna have a great time yeah yeah yeah yeah stick to the three to six bill quantize quantize quantize quantize and uh and then colab like we we teach entire courses in collab and we do a ton of fine tuning throughout so you know just try to be as efficient as possible don't sit there and do tuning for you know days and days at a time if that's not really something that you're interested in you know use small, try to make the model as small as possible through picking the small size of Hugging Face and then quantization for sure. But yeah, there should be nothing stopping you if you're a beginner. You don't have to get AWS. You don't have to get all these things. Okay, Islam, we got a question that's getting upvoted here. Can I do this fine tuning with Lama CPP? And is this fine tuning possible to plug into the end-to-end fine tuning within a RAG framework? So E2E fine tuning within RAG framework, yes, 100%. The RCAI, we've done an event with them. Their DOM framework and GitHub, we'll get a link for you guys to drop into the chat. That is 100% a great tool that does leverage or can leverage LoRa as well as quantized methods. In terms of Lama CPP, I'd have to double check. I don't know off the top of my head, but I will double check and then we can include that information in a comment if I'm unable to find it before the end of our time together today. Okay. All right. Back to Mandy's next question. We say weights and biases when we talk about ML models or neural network models. So if weights are parameters, are we saying weights and biases that are parameters in the LLM world are weights and biases parameters? Let me think through this question. world are weights and biases parameters? Let me think through this question. We say weights and biases when we talk about LLM. So if weights are parameters, are we saying weights and biases parameters? Like our bias is also parameters? I guess is that the question? No. But yes. I mean, I mean, at the end of the day, the thing we care about is the weights. That's, that's, that's, that's all answer this question. We want to update the weights, aka the parameters. Okay. All right. Good stuff. Then I'm gonna go ahead. Last manny question here. Can you speak about examples of LoRa adapters? Like, what are they? And what are they created for? a tool perspective. So let's say we create a LoRa adapter that's very good at translating natural language to SQL. And then we create another LoRa adapter. And that LoRa adapter has been fine tuned to translate natural language to Python. Then we create another adapter and you you see you can kind of go on that the idea is that whenever we do inference we can choose whichever of those adapters or those laura layers to flow information through that's going to make our output consistent with what we fine-tuned it to do so you can you can think of them as little hats you can put on your model that's going to change its behavior, but it doesn't touch the, it doesn't modify or it doesn't have to modify the base model at all. Just kind of this hat that sits on top of it, but gets it to do a different job. And the idea is that we can choose those hats as we want, even at time of inference, we can choose which hat we want it to wear. Yeah. Yeah. And I mean, you know, this is like the thing for businesses too. It's like, if you think about these adapters, man, it's like they're plug and play. And so if you want the LLM to do something super specific, that prompt engineering has only gotten you so far and you just can't get what you need exactly to get in and out in specific ways with your customer or your user. If you want to really constrain what your user can put in, you want to really constrain what comes out, this fine-tuning piece, this lore adapter piece is going to be like your friend. You know, we had a great meme that we posted on LinkedIn recently where it's sort of like if you're doing fine tuning, you're kind of doing LoRa. So it's sort of like this is a big question. You know, examples of LoRa adapters would be like anything that you fine tuned, you know, you might say and. OK, we've got a couple of minutes left. I'd like to shout out out to you know thanks for the great note just want to appreciate your efforts uh appreciate a lot it looks like we've got uh george i think he's struggling with a specific error maybe we can comment on that after the the event he's he's put his error into slido as well um i guess uh last question this is a big question. So you can take maybe two minutes, Chris, what are the trade-offs of using dimensional reduction techniques like LoRa, QLoRa, PEFT on LLMs in terms of training, inference, fine tuning? Like when you think of trade-offs, maybe best practices here, what do you think of? I mean, the big one is quality or like how good the output is uh there is a trade-off there it's really small and beyond being really small it's really small like so okay this is this is the way i think about trade-offs when it comes to laura and and the crew uh i can i can find you the laura model right to be let's say like 98% as effective as full fine tuning, right? But I can do that in a 10th of the time with a thousandth of the resources, right? So divide by a thousand, the number of resources. I mean, that is a trade-off. There is a trade, you're losing 2%. But like, it doesn't feel like a real trade off. And especially in terms of business value. It's not like a, it's not a real trade off these days, like, especially if you use a high enough R or rank in your your Laura, so you're using that kind of 128 are, you're still getting a massive reduction in compute but you're retaining so much of the performance that it it truly doesn't feel like a trade-off it there is a trade-off to be clear there is always technically going to be a trade-off but it lets you do things you wouldn't be able to do so it doesn't feel like a trade-off i I mean, for small companies, you can fine tune a model that does a whole new novel thing that fuels your business, that is your business, right? That you just couldn't do if you didn't use these methods. In that case, there is no trade-off, right? It's enabling you to do something that was previously impossible to you. That's only advantage. When it comes to inference specifically, possible to you that's only advantage uh when it comes to inference specifically both uh the the Q Laura or any quantized uh method using bits and bytes and Laura if you're talking about non-merged Laura adapters do impart a small inference latency penalty it is. At scale, it can maybe be felt, right? If you're really getting to those hundreds of thousands of requests per second compared to a very efficient model, you might want to re-quantize that to another format and serve that model directly instead of having it part of your LoRa stack. But again, these are problems that come with scale and that scale kind of also helps you fund the solution. But outside of that, you're not going to feel these issues until you're into the six figures or more requests per second for your kind of LLM stack. So I would say there are trade-offs, but when you're getting started, they really don't appear as trade-offs. All right. Yeah. Okay. So use PEFTQ, Laura, unless you got a million requests per second. Sounds like a plan, dude. All right. Cool. Let's go ahead and wrap it up. Thanks, Chris. And can't wait till next time. Thanks, everybody, for joining us today. Again, next week, we'll be back talking inference and serving and how to do it efficiently with VLLM, one of the hottest open source tools out there for doing that. We'll tell you a little bit about the tool and its background. If you like this session, you might also really like cohort four of LLM Ops, LLMs, and Production launching February 13th. In that course, which we're going to be soon announcing an expanded curriculum for, you'll learn to prototype and scale production LLM systems, including using RAG techniques, including fine tuning, and so much more. Check it out in the link. And then lastly, please share any feedback you have on today. You can drop it in the chat or you can drop it in the feedback form. That will drop to you now. And that's it for today. Until next time, keep building, shipping, and sharing, and you know we'll be doing the same thing. See y'all next week.
Fine-tuning with QLoRA (Quantized Low-Rank Adaptation)
​GPT-4 Summary: Discover how to supercharge your LLM application development by mastering quantization, a game-changing technique that dramatically reduces the size and computational demands of large language models (LLMs). In our upcoming live event, we'll dive deep into the essentials of quantization, demonstrating how it makes LLMs more accessible and cost-effective for tasks like loading, fine-tuning, and inference on limited hardware. Learn the ins and outs of using the bitsandbytes library to load quantized LLM parameters for our Mistral-7B demos, and explore advanced fine-tuning techniques with QLoRA, building on the principles of Parameter Efficient Fine-Tuning and Low-Rank Adaptation (PEFT-LoRA). Whether you're working on development or in production, this event is your key to speeding up the LLM application cycle. Code will be provided, ensuring you have everything you need to implement these strategies effectively. Don't miss out on this opportunity to elevate your LLM projects!

Event page: https://lu.ma/quantization

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/7CrWMfvZg2NXWh6aYsKkfr

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/greglough...

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our next AI Engineering Bootcamp on Maven today! 
https://maven.com/aimakerspace/ai-eng-bootcamp

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/u63yUJRD9AijuTE98