 Hey Chris, is it true that we can use PEFT-LORA to train less than 1% of the trainable parameters in LLMs and still get great results? That's absolutely right, Greg. So how much data do we need to make that happen? It's a lot less than you think you would need. So you're saying that just with a little bit of data, a little bit of fine tuning, we can mod the behavior and performance of LLMs to be efficient for our tasks, our applications, and our businesses. That's absolutely correct. Yes. Dude, that's so cool. I am pumped to dig into this today. We'll see you back in a bit dude for the demos. Hey, everybody. My name is Greg Loughnane, and I'm the founder and CEO of AI Makerspace. Thanks for taking the time to join us today. As we kick off the new year, we want to demystify one of the most mature aspects of building with LLM applications for you today. And that is fine tuning, parameter efficient fine tuning, and low rank adaptation or LoRa. And this will hopefully help you really get started on the right foot in 2024. We'll be collecting questions today via Slido. So please drop your questions directly into the link in the chat and also upvote your favorites. Of course, Chris, my friend, CTO of AI Makerspace and the LLM wizard will be back to lead our code demos soon. So let's get right into it today. Let's talk efficient fine-tuning of LLMs with low-rank adaptation or LoRa. As always, we want to make sure we align our aims for the session. If you're joining us, this is what you will get out of today. What is fine tuning? What is PEFT? What is LoRa? How can we actually fine tune these LLMs? Let's see exactly how it's done with the latest and greatest tools from Hugging Face, including one of the latest and greatest open source models. So to contextualize this a bit, models. So to contextualize this a bit, first we want to talk about as we build LLMs, why this idea of fine tuning is important in the first place. Then let's talk about fine tuning proper and then dig into this PEFT, this library, this idea of doing this a little bit more efficiently. Finally, we'll talk LoRa or low-rank adaptation of LLMs before we see Chris back to lead us for some demos. And then, of course, any questions you have, we'd love to answer them. All right, so when we prototype, when we're building those LLM applications in 2024, right now, we generally think, okay, we want to prompt engineer as far as we can. We want to kind of get this thing up and running in a minimally sufficient one-shot, few-shot type of way. Then maybe we want to sort of ground our LM application in some of our own data, some of our own context. This is where retrieval augmented generation comes in, as well as, you know, sort of just the idea of this question answering system. And finally, after we do some prompt engineering, generally step two would be doing some RAG. We're often looking at fine tuning with a keen eye and trying to understand when fine-tuning becomes of use to us. A lot of people are asking, do I do RAG or fine-tuning? And the answer is, well, it depends. It's not always linear like this, as OpenAI has sort of put out there. There are actually two different things that we're optimizing when we move from prompt engineering into RAG or fine tuning. And the fine tuning piece in particular is what we're talking about is we're talking about the way we want the model to behave, the kind of inputs we want it to get, and the kind of outputs we want it to provide. Of course, general large models can have general inputs and general outputs, but more specific task-oriented models like the ones we'll need in our businesses, they're going to be a little more dialed in, a little bit more fine-tuned. And oftentimes, as you build these applications, you're going to have to go from prompt engineering to RAG to fine-tuning and back again to get this thing to a human-level performance that you're happy with to first deploy it to production. And then once you're in production, of course, continuing to dial in the RAG systems and to fine-tune the LLM as well as the embedding model is going to be very important as you try to serve more and more users, stakeholders, customers better and better. It's also worth noting that as we head into the new year, one of the things that's gaining a lot of traction right now is this idea of small language models. Here we see this word efficiency that we're going to see a lot today. Microsoft put out in the State of AI report, this was noted, last year, a couple of papers, and one of them was called Tiny Stories. And it was sort of interesting because it kind of compared the 125 or so million parameter models like GPT-NEO or GPT-2, the small versions, that really don't do a very good job at generating coherent text sort of out of the box to a much smaller model instead of 125 million parameters, just 10 million parameters, but trained on synthetic data that was made up of stories using very, very simple language, the kind of language you could talk to a three or four year old with. And the tiny stories data set showed that it was possible to generate smaller, much smaller language models that really gave some pretty solid English generation. Whether you're training at 10 million parameters or you have a few more parameters but you just have one transformer block, this was shown to be possible. Also when we got specific about just coding, that's where this other paper, Textbooks All You Need, came in, where instead of training a huge model and then trying to make it good at coding, it was simply taking a relatively small model, 1.3 billion parameters, still very, very large in the context of all models we've ever seen. But when we think about the leading models of today, there are at least 7 billion or so. And fine-tuning that and fine-tuning that on, quote, textbook quality coding exercises, where there was some taken from textbooks, some synthetically generated, and really produced a very small, relatively speaking, only 350 million parameter model that produced great results. These are worth highlighting because whether we're talking about training from scratch or we're talking about fine tuning, we're kind of talking about the same thing and we're kind of talking about dialing these models in for our use cases. And that's exactly what fine tuning is all about at the end of the day. So when we talk about fine-tuning, as in the latter case of textbooks are all you need, we're talking about taking a language model that was trained using unsupervised pre-training, probably some combination of instruction tuning and other alignment techniques as well, although not in all cases. So we're taking that base model that we get from this initial training, and it could be a chat model again, or an instruct tune model, but then we're kind of taking our specific instruction examples. and those might be very specific. They might be generated based on our specific product line. But those examples are the ones that we then take and we update our language model to be very, very good at, producing this supervised, fine-tuned model that's really more dialed in to the kinds of inputs and outputs and specific instructions associated with what we want the model to be good at. Because of course, these models can be good at so many different things out of the box. So as I alluded to just a moment ago, whether we're talking about fine tuning or training, we're talking about the same thing. We're talking about updating the weights or the parameters within our large language models. And this is not new to anybody that's been around the game for a while. I mean, training a simple, deep learning neural network is not fundamentally different than what we're doing when we do fine tuning of a large language model, although there are some key differences that we'll highlight today. The mechanics are there. We have inputs that we pass through the weights, and those weights fundamentally are the things that are going to be updated. They can be updated through back propagation, like the classic technique. We're also seeing some really interesting things happening out there in the space where you can do fine tuning without even using back propagation. And so expect a lot more innovation in this space, especially as small models become more and more popular. Chris did a really great example of how to talk through Laura at an even slightly deeper level. He had a bit longer form opportunity on his YouTube channel that I highly recommend checking out if you want to dig further in on the specific visualization of weight matrices here. But we're going to today sort of go from this mechanics of fine tuning into this idea of PEFT. I mean, this is a great meme because it's so true. This is sort of like the new AWS mistake, right? Is do I really need to fine tune this like that again? Do I really need to spend this much compute messing with things? And this is where we get into this, let's be a little bit more efficient space. PEFT is parameter efficient fine tuning. And the big idea, the problem that it solves is when you fine tune on downstream tasks specified by those data sets of specific instruction and output responses, you can get huge performance gains compared to off the shelf LLMs, zero shot, right? So zero shot is not really providing a ton of specific instructions, not providing a ton of examples. So you can get pretty far again with prompt engineering. And so when this fine tuning comes into play, that's going to be a judgment call for you as you continue to build and prototype your LLM applications. And when we think about why PEFT, well, we've kind of covered it already, but LLMs are so big and billions and billions and billions of parameters are hard to deal with. They're hard to load into memory. They're hard to do inference on. They're hard to just manage. to sort of leverage the best parts of them but not do it in such a way where we just can't ever get it done on the machines that we have or the cloud instances that we have access to because this problem isn't going away as we've seen these models although they will continue to get smaller and smaller things like mistral and others are are pushing those boundaries on the OpenLM leaderboard all the time. We are going to see these models get bigger and bigger still as we approach and move towards AGI. Fundamentally, we want these models to get bigger and bigger. We want to see what they're capable of. So the biggest labs will be generating bigger models. Some of the startups out there like Mistral will start to see generating smaller models. But fundamentally, we're talking about these weights of the trained model, the parameters in the trained model. When we download a model, we're downloading the weights, we're downloading the parameters. This is the same thing and those parameters are always are always represented by a certain number of bits and generally if we download in full precision we're dealing with a lot of parameters this gives way to the idea of quantization which we'll cover very soon in an upcoming event next week and we'll see a little bit of quantization today when we see Chris's code demo. But it's important to understand that we're just dealing with these weights, these parameters, and it's important to find ways to be more efficient when dealing with them. So when we think about training those neural networks, training those parameters, we need to think about it's the same as fine tuning, but it's not all exactly the same. Because when we have a pre-trained network, those weights are pretty solid. We're not going to change every single weight a whole bunch when we do fine-tuning. For instance, in this simple example of sentiment analysis, if you were trying to do some fine-tuning for a specific domain, you imagine many of the weights within the network wouldn't be particularly different, although some might be. And this is how we want to think about fine-tuning. We're not changing everything. We're simply fine-tuning. And there are two challenges when we do fine-tuning. When we do full fine-tuning, even though we don't necessarily have to change every weight a whole bunch, we're changing all of the weights. This is very hard to do on our own hardware. And again, when we do inference and we try to store and deploy these things efficiently for low cost, it's very, very hard to do. We need smaller models. And so the solution is leverage a smaller version of the large model by only dealing with a smaller number of weights and then only dealing with a smaller number of trainable weights when we do fine tuning. This is exactly the approach we'll take today. This allows us to have models that are better not only at out of data set pieces of information we want to do inference on, because fundamentally they're more general, but also allows us to do really a whole lot with not very much data, which is very, very cool. So from PEFT to LoRa, this image probably doesn't mean a whole lot to you if you're not familiar with LoRa yet, but after this brief discussion I think it will. LoRa as sort of an entry point for those of you that are complete noobs to this is simply the number one parameter efficient fine-tuning method that you should go out and learn today. It's the first one to start with. From there, you can start to dig deeper. But what's really going on in LoRa and what's the intuition behind why it works? Well, the intuition for LoRa is similar to the intuition behind why fine tuning works. We have these pre-trained models and not everything within those models really matters the same amount. They have a low, quote, intrinsic dimension. That intrinsic dimension, intrinsic simply means essential, that natural dimension of things that matter when it comes to the behaviors we want the model to exhibit is what we're really focused on. So we can actually change not every single parameter a whole bunch, but we can change just those key parameters. And of course, identifying those key parameters is science unto itself, one we expect to see a lot more progress in the space in 2024 on, but the sort of key intuition and the idea is that we don't need to change all of them a whole lot all the time. When we talk about LoRa, we move from this idea of intrinsic rank. And LoRa is really focused on specific matrices within a transformer architecture. So the hypothesis that the authors put forth here is that the updates to the weights also have a low, quote, intrinsic rank during, quote, adaptation. So we have this idea of intrinsic rank and we have this idea of adaptation. So let's talk about rank first. Well, hearken back to your linear algebra days in school and maybe you recall that rank is the number of linearly independent columns in a matrix. Okay, that's not very helpful. If we think about it as a weight or parameter matrix, maybe it's slightly more helpful, but still not super helpful. What we want to think about is these linearly independent dimensions are the ones that provide unique information about what we need to solve the problem we're solving to exhibit the behavior we want to have this LM exhibit to have that task be completed properly. This is a screenshot of the Word2Vec embedding model shown in three dimensions through principal component analysis, but there's a 200 dimension underlying representation at play here to put each of these words relative to one another. So each of those 200 dimensions, you might say, matters. The question is how many dimensions matter in your language model and for your task? And this is sort of what we want to ultimately drive towards in this small language model domain. Now, as we think about how LoRa works beyond this idea of matrix rank, we want to think about the two-step. Step one is for it to freeze all pre-trained weights. This is cool and this is the sort of adapter piece. Then we want to sort of be able to create these adapters, these injectable, trainable, rank decomposition matrices that we can plug and play into our pre-trained model weights, into our initial LLM that we downloaded. And so these sort of plug and play adapters are made up of these rank decomposition matrices. Again, if you go back to school, talk about linear algebra, decomposition of matrices is just about chunking big problems into smaller ones to make things more computationally efficient. If you guys ever use MATLAB back in the day and just tried to invert the million by million matrix real quick you can definitely make matlab go burr and it's pretty fun but there's better ways to do this we don't try to invert million by million or billion by billion matrices and we don't try to do really really computationally intensive things with matrices when we're dealing with llmsMs either. This is where LoRa comes into play. Finally, we want to understand that LoRa is going to be used in a transformer context. It's not simply a feedforward neural network, although it's helpful to use that image as we get a handle on it. We did a really long form deep dive on transformers. We'll probably do another one soon. So if you're interested in that, definitely check that out. It's transformers we'll probably do another one soon so if you're interested in that definitely check that out it's uh transformers for a couple of hours there and then the important point here is that laura is not changing everything within the transformer laura is associated simply with the attention layer in each of the transformer blocks. So it's not associated with the attention layers and the feed forward layers. It's associated with simply the scale dot product attention that's happening within each of the blocks. And as you'll see, it's not even just, it's not even all of this. It's just the queries and the values, the Q and the V, that we're actually leveraging the LoRa adapter for. And of course, if you're familiar with transformers, you already know that this is an encoder block and this is a decoder block. Many of the common LLMs you'll deal with today are decoder only style. And so that's worth noting here as you start building with decoder only LLMs and leveraging LoRa for fine tuning as well. So LoRa provides a ton of advantages for us. It provides a more efficient way to train fewer parameters. Those adapters are very plug and play. If we simply keep the initial weights and then train a bunch of adapters for different tasks, we can use the same model for a whole bunch of different things. So you can start to get sort of curious about how this might dovetail into hosting and inference and doing a little bit more efficient work on the ops side when you start to deploy more and more of these applications for your company and then you can combine them with other methods you get pretty good results and then a key point here is that pretty good results and then a key point here is that you can merge the adapter weights with the base model and in doing so and this is a an awesome sort of hugging face library thing you can make sure that there's no additional inference latency versus the base model itself. All right, so to recap before we get into the demo today, we talked fine tuning. And when we think about fine tuning, we want to think about simply modifying LLM behavior by updating weights or parameters. We talked about PEFT, which is fine tuning with less weights or parameters. And we talked about LoRa, low-rank adaptation, which is fine-tuning with factorized or decomposed matrices. So, if we put it all together, we have PEFT LoRa fine-tuning, which is just modifying LLM behavior by updating less weights or parameters with decomposition and factorized matrices. So, in a diagram, you can kind of think about PEFT-Ra fine tuning versus regular fine tuning a little bit like this. So today's build is going to be a classic application the UNO reverse card application given a response an LM output we're going to predict the instruction or the prompt or the LM input we're going to pick up one of the prompt or the LM input. We're going to pick up one of the best models off the shelf that we can find that works well with this fine-tuning paradigm, and that's Mistral 7B Instruct V0.2. Definitely one of the best off-the-shelf models you can pick up today. And the fine-tuning data're gonna use is we're gonna use the Alpaca GPT-4 dataset where we're simply going to switch the output and the instruction. You can check out more about this dataset at this paper right here. But without further ado, it's time to get into the demo fine tuning with PEFT Laura. Chris, the LM wizard is back. Let's see him in action. Thanks Greg. Yes, okay, so the idea is pretty simple. What we want to do is we want to do exactly as Greg has outlined here. So, you know, this is a method to fine tune a model on a much smaller hardware than you should normally be able to get away with. Now, you will need to use 40 gigs of GPU RAM for this specific example. During training, it does peak pretty high. So if we look at our GPU RAM, you can see we used peaks of 15. So the T4 might not be big enough for you. You can use a very small batch size, of course, and that will help you kind of cheat around some of the space requirements. But for the most part, we're going to, you know, we're going to focus on using a larger model. Okay, or a larger card. So we're going to use three main libraries here, that's going to be PEFT, transformers, and bits and bytes, we're going to kind of gloss over this bits and bytes step, though it is extremely important. We're going to talk about gloss over this bits and bytes step, though it is extremely important. We're going to talk about that more next week when we talk about quantization. But for right now, we're just going to focus on PEFT and then transformers. So the idea is, you know, we have this model, Mistral 7B Instruct, but we want to make it better at a specific task, right? So we're not trying to teach the model anything new. We're just trying to, you know, get it into a place where it's doing our behavior that we desire a little bit better. So we're gonna use the Mistral 7B Instruct V02 from Hugging Face, as well as we're gonna use the Alpaca GPT-4 dataset. So you can just find this model on the hub right here. And then you can find that data set also on the hub right here. Perfect. All right. So what is PEFT, right? PEFT is parameter efficient fine tuning. Like Greg said, all that really means is that we're fine tuning with less parameters or more efficiently leveraging our parameters. our matrices have inherently or intrinsically low rank uh you know this is discussed in in the papers uh that that you saw greg talk about and so we want to fine-tune our model taking advantage of that kind of trick and so uh the way we set this up is pretty straightforward it might sound complicated uh from the the kind of concept side but from the code side you know Transformers Library Huggy Face has us uh as us uh ready to rock so we're just gonna grab our dependencies and then we're gonna go ahead and we're gonna load our model uh you'll notice that we're loaded loading the Mistral AI Mistral 7B instruct v0.2 as we discussed all this stuff is great basically what this is doing and we'll deep dive again next week but discussed. All of this stuff is great. Basically what this is doing, and we'll deep dive again next week, but the idea is all this stuff is doing right here is loading your model in a very small fashion, right? So we're going to massively reduce the number of bits that each of our weights are going to take, or each of our parameters is going to take up in memory. And that's going to help us load it into a smaller card. So you'll notice that we have this whole model trained and we only peaked at 15.4 gigabytes of GPU mem, which is pretty awesome. After that, we just load our model. Kind of normal stuff, right? Now, we have to do this kind of tokenizer processing. The reason why is we're using a Lama tokenizer as a root. You know, all of the mistral suite of models is kind of like a redux of the Lama model. So we want to make sure that we're doing this pre-processing so our training goes smoothly that's the only reason that's there now when it comes to our model architecture you'll notice that we have these 32 decoder layer blocks each of them has a self-attention layer with QKV proj and then our output as well as some rotary embeddings. You'll also notice that we have this MLP, which is gate up, down, and our activation function. And then we have some layer norms that are also useful. The big thing to focus on here is these attention layers. Now the original LoRa paper, when it came out, kind of talked about how we want to target, you know, say Q and V as a rule. There's been some more information that's come out that says it might be prudent, especially when using things like QLoRa to target all of our model or all of our layers equally. So all of our non-layer norm layers. But for this example, we're just going to use the automatic PEFT version. So what this means is we're going to let PEFT make that decision for us, though you could see increased results if you did target more layers or more modules to be more specific. We're just going to focus on Q and Vproj today. So now that we've got our model loaded, we've kind of decided, you know, we've decided what we want to actually focus on. We need to turn our model into a format that's compatible with LoRa. And the way that we do that is pretty straightforward. We're also gonna use this print trail parameters, helper function just to really see how few parameters we need to use. So the LoRa config has a number of options. The first one we see is R or rank, right? So when we talk about low rank adaptation, this is the rank we're talking about. Now you'll notice that even though we have a low rank, we can go pretty high with it, but we're not going to get as high as say our base dimensions. So in this example, we'll be using 64. No particular reason. R is a hyper parameter. You'll want to do some kind of hyper parameter search in order to determine the best version of R. But for right now, we're just going to go ahead and we're going to use 64 as a example. as a example. Now, the idea of this rank, right, is comprised of the fact that we're gonna have this base model dimension by R matrix factored with this R by base model dimension sub matrices. So we're gonna build these sub sub-matrices, which are going to be smaller, and that's what we're going to update, right? Because as we'll see in the architecture, what we really want to do is we want to compute the delta weight or delta W, right? And the way we do that very efficiently is we only care about this factorized version of our full weight matrix. So let's look and see what this looks like in the code. You'll see that we just pass in R, 64. That's great. And then we pass in our lower alpha, which is 128. Now, the kind of conventional wisdom here is that alpha should just be twice R. There's an incredible blog post from the Lightning AI folks where they did many different experiments. This is not always true, but it's a great place to start. You might want to explore different ratios, but for the most part, your alpha should just be set to twice what your R is. And alpha is just a scalar. Now, for target modules, again, we kind of have a decision here. We can let PEFT take the wheel, right? If it's a model that's already existing in the PEFT library, or we can target specific modules. For this instance, we're just going to let pef take the wheel, and it's going to choose K, or Q and V, and it's going to omit K. But we could manually add K as well as whatever MLP layers we wish to add as well. Totally up to you. We have this dropout. Dropout is the important thing to prevent overfitting. We're not using a bias and our task type of course is causal lm since we're doing a causal lm task next we need to prep our model so the first thing we're going to do is prepare model for k-bit training um this is great we want to do this and then we're going to get our peft model also great also we want to do this there's a lot going on under the hood here. But what it boils down to is just moving the model into a format that's compatible with our desired training. Then we're going to print our trainable parameters just to see. You'll see we go from a lot of parameters, right, to 2.7 million parameters. And that represents 0.72% of our total trainable parameters. So we are only training 0.72% of the possible parameters we could train using this method. That's why it's parameter efficient, right? This is how the idea of parameter efficiency comes in. Despite the fact that we're training on less than a percent of the total trainable parameters, we are able to get great results. So let's look at the model architecture to see exactly what's happened here, right? So we have our Qproj, which was previously just a linear. It's now been replaced by a LoRa linear, which has our base layer. That's that W, right? And then it has our LoRa dropout. Then it has LoRa A and B. LoRa A is the 4096 by 64, and LoRa B is 64 by 4096. If we were to combine these two matrices, we would get a 4096 by 4096 matrix, which is what we would desire since the input and output is 4096 by 4096. So the idea here, right, is that we're just breaking down our delta W into these two sub matrices. And all we do is every time we go through and we calculate what our weight changes should be, we update LoRa A and B, and then we combine them together, and then we are adding those, literally adding those to the base layer to keep track of our changes so that we can keep learning. That's it. Both of these are initialized. One is initialized as a random Gaussian, and the other is initialized as zeros. There's some intuition behind why we do that, but for the most part, that is the rule and that's what's going to happen. You can do kind of like more tactical initializations, but the base methods are just going to use the actual Gaussian and then zeros. You'll notice that K proj is still just this linear and then V proj also has ourRa layers. That's because it only targeted v and q. And that's it. That's all LoRa does, right? So now every time we update our weights, what we're really going to update is these LoRa sub-matrices. And then we're going to combine and add them to our base layer in order to get the final results. So how do we train this on data now, right? It's great. We kind of have a better understanding of how it might work, but how do we actually train on data? Well, it's the same as every other time, right? We're going to grab some data. We're going to create a data set. It's got 52,000 rows. We don't need all 52,000 for the sample. We're going to look at how this might look. So we have some instructions. We have an input. We have an output. That's great. We're just going to select a subset of 5,000 of these rows. And then we're going to convert it to this kind of format, right? Generate a simple instruction that LLM could use to generate the provided context. Then we're going to provide the context and then the response. The idea being that, you know, we want to, in this case, show the LLM the response and have it create the instruction, right? So this could help us to synthetically generate an instruction data set. to synthetically generate an instruction data set. For generating our prompt, this is a function that a library we're about to see is going to use to help us understand what each prompt is. So we basically feed it a row of the data set and it generates a prompt in this format. That's all this is doing. Now we'll be mapped over our entire data set and it will result prompt in this format. That's all this is doing. Now we'll be mapped over our entire data set. And it will result in something like this. Create a simple instruction that could result in the provided context. Then we have the instruction flag, and then we have a bunch of context. Then we have the end instruction flag, and then we have our response. And that is it. Once we have that set up, we can move to creating our training arguments. This is all boilerplate. It's a lot, but it's all boilerplate. Basically, we're going to have our output directory. We're going to train for 100 epochs. We use a batch size of four. If you're doing this in a T4 instance or the free collab version, you might want to reduce this to two or one in order to prevent out of memory issues. We're gonna use this gradient accumulation step. This is kind of a way to cheat out a higher batch size. We're gonna use gradient checkpointing as well as the paged AtomW32-bit Optimizer. This is straight from the Qlora paper, which is just a good optimizer. This is straight from the QLORA paper, which is, you know, just a good optimizer to use. We're going to have a fairly aggressive learning rate. And then of course, we're going to set our D types. We're going to talk about this more next week. But for now, the idea is we want we need to be able to ensure that we're computing things in the correct D type. And then we have some remaining kind of boilerplate. Again, these are all hyperparameters. You will want to fiddle with these if you're producing a model for production use in order to ensure that you have the best possible result. Then we're gonna use our TRL SFT trainer. SFT trainer just means supervised fine tuning trainer, right? Easy peasy. We're going to pass in our model, our data set, our LoRa config, pass in this max sequence, which is just going to be 2048. You can set it to whatever you'd like, as long as it is less than or equivalent to the max sequence length of your model. We're going to pass in our tokenizer, and then we pass in that generate prompt function we built above so that it can be mapped over our data set. And then we're going to pass in the training args that we just set up above. After that, all we got to do is everyone's favorite thing to do, which is.train. You'll notice that the loss slowly goes down over the course of training. So we're, you know, we get down to 806 and then we're down into the double zeros. So that's great. It does plateau fairly quickly. And then we just kind of keep training. So likely this is overfit. We're going to go ahead and save this model. Now you can't really, it's interesting, right? You can't really run inference with just this, right? So this model is a LoRa model. We'll need to set up a auto PEFT model. Now that PEFT model is going to leverage the adapter that we, or the, the, those weight matrices we wound up training, right? So you can't run inference on the newly trained model. You have to convert it to a format that is suitable for inference. Now you could either merge the weights that you trained in, which is a basically, you know, you take that delta W, that's the combination of those two smaller matrices, and then you just plop it on top of the base layers. Or you can do that process during inference. Now that last sentence, right? Do that during inference is what makes LoRa a absolutely insane approach to use in production in terms of the flexibility, right? So you can fine tune the same model on a thousand different tasks. It's just a little bit hyperbolic, but the idea is you can find you on a number of different tasks, right? And then at inference time, you can decide which model you use. So you only ever host the base model, and then you get to choose which combined matrices you use at time of inference, which is what makes this such an incredible tool for a production environment and such an incredible tool for hosting models. And all you have to do is use this auto PEF model for causal LM. And then it's going to load up that model with the, with the additional PEF adapters. We use the word adapter here loosely, because there is a method called adapters. This is distinct from that, but it is the same idea. It's this entity that we keep track of that kind of is what we train, right? The same way that you download model weights, you can download your LoRa weights. So let's look at how it did. So we generated a simple instruction that could result in the provided context. And the context is the odd one is Telegram, Twitter, and Instagram, social media, blah, blah, blah, blah, blah. And it says, identify the social media platform that is not an instant messaging service. And then it lists three options. And that is a very much more verbose, but it is absolutely the kind of instruction we were looking for. Now, how did the base model do? So it's great that the fine-tuned version did well, but how does the base model do? Well, the base model does pretty poorly. It says, it identifies the platform that's primarily used for instant matching and voice over IP. Instead of sharing information, the OO is Telegram. So it doesn't do a terrible job, but it definitely does a worse job it doesn't provide the uh lists of options and so therefore how can you identify the odd one out in any case the idea is even with only 100 epochs in collab only using a maximum of 15 gigabytes of VRAM, we're able to fine tune a 7 billion parameter model to be better at a task. You know, this whole process took very little time and we couldn't choose which of these fine tunings to use at time of inference, which is a, cannot be overstated how powerful it is. But that's all for me. So with that, we're going to pass it back to Greg and, uh, I'll see you guys in a bit for some Q and a man, Chris. Thanks so much. That was deep and super relevant to anybody building those production LM applications this year. Love to see how much of this is really one-to-one from prototyping into production in case you guys missed it we just trained only 2.7 million of a possible 3.8 billion trainable total parameters and that's less than one%, 0.72%. And just to kind of try to put this together a little bit in your sort of matrix algebra, linear algebra minds, remember we're training only the attention layers, right, within the transformer, and we're training only Q and V. So Q has dimensions, same as the the attention layer 4096 by 4096. So we had Laura A and Laura B of these sizes. Remember Chris chose 64 dimension from the paper. And then for V, it's got a dimension 4096 by 1024. So Laura A and Laura B were slightly different sizes so this is kind of how all this comes together again those adapters can be plugged right in even at inference too that's really cool and so you know in conclusion we got some great results and we learned that PEFT Laura fine tuning is all about that modifying LLM behavior by updating fewer parameters with those factorized matrices. Fine-tuning is going to be a heck of an important skill in 2024 as it helps us leverage only the essential, those intrinsic dimensions for our downstream tasks. And as the age of small language models comes upon us, it's gonna be one of those things that we're asked to do more and more. As we've mentioned a couple of times, next week we are going to cover quantization and QLORA. We'll talk about things like bits and bytes, and we'll talk about quantization in general and the trends towards smaller and smaller models. We'll continue our discussion of that is not going anywhere in 2024. So with that, love to open it up for questions and Chris, come on back up and let's get it rocking and rolling. Feel free to drop your questions in the Slido or in the YouTube chat and we'll do our best to get to them. Chris, that was awesome, man. I think they liked it too i love laura it really does the thing it does the thing yeah it's uh it's it's so interesting i think it's one of the few things that has kind of stuck around the vast majority of last year we didn didn't go anywhere. Nobody budged it, except the same people that kind of came out with it, but kept coming out with better stuff. So, you know, that's what we'll talk about next week. So Deb asks, could one use LoRa to fine tune a very large image recognition model so that it recognizes a few cases of interest on which they have not been trained? Yeah. So you should, I mean, so LoRa is not only applicable to language models. It does see a lot of uses in models like kind of a stable diffusion-esque models, so image generation models. I would assume that you could do this. I haven't. And so I don't want to say super concretely how effective it would be. But I mean, you could apply LoRa to basically anything. apply LoRa to basically anything. It is not specific to any architecture. You know, that's why we can apply it both to the attention weights as well as the MLP weights with with little, little issue. Yeah, yeah. So that idea of finding new stuff it hasn't been trained on maybe maybe just if it's sort of the infinite object detector already, sort of why not kind of thing. Yeah. So one of the things you can do is you can fine tune object detection algorithms to recognize better or certain classes more robustly. And I'm certain you could use LoRa to achieve that. Okay. Okay. Yeah. So try it out, Deb. Let us know for sure. David Reitman asks us, asks you, to define great results with PEFT, I believe you said. And we got great results. As one would assume, if you're only using a subset there, there's some sort of loss on the output quality? Is there? Ah, yes, but also no. on the output quality is there ah yes but also no uh so yeah it i would say for the most part laura is uh it comes kind of without a specific performance hit uh it will it will compromise your model's ability to do kind of, all of the tasks it originally was able to do a little bit, but in terms of like the base functionality of a language model, which is generating text, there's no real performance hit. So, if you had a model that was before good at math and prose and technical writing, let's say, and you used LoRa to make it really, really good at technical writing, there is the possibility that it might be worse at the math, right? But it's not going to be much worse. That's what was found. And that's, again, due to the fact that these behaviors exist in a low dimensional space that we can manipulate very readily and easily. Nice, nice. Okay, Ali asks, should we always use a fine-tuned model instead of prompting for production-ready applications, considering that accuracy plays a crucial role? Should we always be fine-tuning, Chris? What do you think? No. There's no fast rule for this. So this is my opinion, but there are lots of times that you do not need to fine-tune, right? In fact, when, when prompting is enough, there's no real reason to fine tune, right? I mean, if you get very good evaluations on whatever you're building without any fine tuning, so prompting, you know, achieves your kind of benchmark targets, then there's no, there's no reason, achieves your kind of benchmark targets, then there's no reason to do anything other than just prompt the model. There are definitely lots of cases where you will need to fine tune, but if you don't have to, then why spend the cash? 100%. Yeah. And we were sort of covering this idea of how to prototype. So you start with the prompt engineering, maybe move to RAG system, and then maybe look at fine tuning. Now, there's some special cases in between, but fine tuning is sort of, let's say, when you're mostly there, right? You know, just fine tune a little bit. And you can get mostly there through some other methods, but definitely it's going to be one that comes up if you're messing with production ready applications this year, Ali. So for students to experiment with these fine tunings and this learning, what would be your suggested, I guess, stack for them to be playing with? What do you think, Chris? It's all in the dependencies in the notebook. That's all you need to get started. In terms of the computing solution, Colab is just good enough, right? In terms of experimentation or like a playground, Colab is good enough. It is, especially if you pay for Pro or you pay for compute units depending on which is a more uh a feasible solution for you and you don't even because what we're talking about inherently right is trading these big models but using much smaller resources then you don't even need like you can use the free version with a batch size of one to fine-tune a mistral seven billion right so it's I would say like collab is a great place to get started it's really easy you don't have to manage the environment it's all done in the cloud and you don't have to like do any weird signups or anything like you don't have to request quotas you don't have to worry about cost while the thing is not running, right? So Colab is a really, really great solution. That being said, running these things locally on a consumer grade GPU, right? Like a 3090 or whatever, or 2080 with 16 gigs of VRAM or even a Mac, right? And Mac just, or Apple just released like a bunch of Mac libraries that make it even easier to run these models on a Mac. Like you can do all, all of this locally if you really wanted to, but if you don't want to fight with environment setup, I would, I mean, stick Colab's great. It's so good. Yeah. Yeah. Yeah. Cause maybe, maybe what the real question is here, it was like, should I be using AWS? Should I be using, you know, should I be using one of these big cloud providers? Which one? And it's like, it's like, well, no, not really. To get started, you really don't need to worry about that. Okay, I'm going to go to the question now from the chat here from Vadi Yala, because I think it's super interesting and relevant to what's happening today. Is it possible to Laura fine tune with the DPO method? We got some lines we can help discern here, Laura versus DPO. You got any hot takes on DPO, Chris? DPO is good. What is it like better RLHF in a way? Not really, not really. But it is, you know, it's, it's a great solution. It's a great thing to do instead of doing like PPO. I would say for the most part, these are compatible systems. You know, you can do PPO with LoRa, you can do DPO with LoRa, you can, I mean, again, LoRa is not specific to a technology. It's just specific to the fact that there is this weight matrix that we know has intrinsically low rank, right? So anytime that's true, you can just plug LoRa in there. In terms of like. DPO versus Laura, I wouldn't think of them as versus or in two separate camps, I would think of them as compatible and synergistic technologies like they are not two distinct things in terms of like you need to choose between them. they can synergize and so that's great. Nice. Nice. Okay. Quick question. Input and output embeddings in the transformers, the encoding and decoding blocks. I said the common ones we use today are decoder only. I did and that's sort of based on this idea that the encoder only style is only used for just a few applications uh today and generally when we see those gpt models we see llama 2 we see mistral we see all the latest and greatest new ones these are decoder only style we've got sort of a long form discussion of this that we've done in the past that we'll go ahead and drop in the chat for you now to sort of dig in to transformers more deeply but that's sort of enough on that question for right now i just want to go to maybe one or two more we got somebody says chris hey i'm quite new here i'm into finance and a sample use case is portfolio management. How would I tune a model to that domain? Finance, portfolio management. Well, there's lots of different ways. I mean, the idea is start with a model that is good at that. So this is kind of the wisdom of our friends at RCAI, right? This idea of using a domain-specific model and then adapting that model. So you might want to use like a Bloomberg GPT or whatever you can get your hands on that's open source, that's equivalent. And then, you know, find your data set or build your data set synthetically and then use laura from there to fine tune uh but the idea is you will get better results if you use a model that's closer to your actual desired behavior remember greg just said it but he said during the presentation and it's so so dang true right we are fine tuning so we're close you know fine tuning only works if we're close, you know, fine tuning only works if we're close to the target, right? If we try to fine tune from here to way over here, it's not gonna happen. So we wanna make sure we start very close to our target and then we fine tune that last few millimeters. Yeah, so I would say start with that and then you can use LoRa, there you go. There you go, yeah. And I would say start with that and then you can use LoRa. There you go. There you go. Yeah. And I would say too, the sample use case being portfolio management is a little bit too generic sounding to me. If you can really dial in which aspect of managing portfolios that you can try to say, okay, this simple task associated with portfolio management, maybe the most annoying task that people have to do. Maybe it's the selection of stocks within even a particular, you know, a particular domain of, you know, the world and of the stock market, like as clear as you can be on the task, that's gonna make it easier to curate the data. That's gonna make it easier to do the fine tuning. That's gonna make it easier to get most of the way there through prompt engineering and rag and and some of these other techniques so yeah great questions everybody thank you so much for joining us kicking off the new year strong Chris thank you so much for dropping the knowledge on us with status we'll see you next time man thank you everybody for joining us live today and this brings us to the end just for today until next week when we talk We'll see you next time, man. Thank you, everybody, for joining us live today. And this brings us to the end just for today. Until next week when we talk quantization and we talk QLORA. Next week, also on Tuesday, January 9th, we'll be launching our LLM Engineering Cohort 2 course. This is our course that teaches everything you need to know to build and train your very own LLMs from scratch, including the transformer, attention, fine tuning, all the different aspects and types of fine tuning, prompt engineering, RLHF. You'll be doing it all and you'll be building your very own LLM from scratch as a capstone. Check it out if you're into that kind of thing or reach out to any of us personally on LinkedIn or wherever you can find us, Greg at AI Makerspace, Chris at AI Makerspace.io. Other than that, please share any feedback that you have on today's event that might help us bring even more value to you in the future. And until next time, keep building, shipping, and sharing, and we'll do the same, everybody. See you soon.
Fine-Tuning Mistral-7B with LoRA (Low Rank Adaptation)
GPT-4 Summary: Dive deep into the innovative world of fine-tuning language models with our comprehensive event, focusing on the groundbreaking Low-Rank Adaptation (LoRA) approach from Hugging Face's Parameter Efficient Fine-Tuning (PEFT) library. Discover how LoRA revolutionizes the industry by significantly reducing trainable parameters without sacrificing performance. Gain practical insights with a hands-on Python tutorial to adapt pre-trained LLMs for specific tasks. Whether you're a seasoned professional or just starting, this event will equip you with a deep understanding of efficient LLM fine-tuning. Join us live for an enlightening session on mastering PEFT and LoRA to transform your models!

Event page: https://lu.ma/llmswithlora

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/cbLiU8BM92VixUStDH24U3

Speakers: 
Dr. Greg Loughnane, Founder & CEO AI Makerspace.
https://www.linkedin.com/in/greglough...

Chris Alexiuk, CTO AI Makerspace.
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for the LLM Ops Cohort on Maven today! 
https://maven.com/aimakerspace/llmops

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/U8oeCWxiWLLg6g678