 Hey, Wiz. So supervised fine tuning and prompt engineering are kind of on like a spectrum, right? That's right, Greg. Yes. Yeah. Yeah. And then like instruction tuning, that's also a type of fine tuning. Is that right? That's correct. And even like the chat style models that we see out there, these chat style LLMs, those are sort of another form of instruction fine tuning written with dialogue chat style instructions. Is that right? All true, Greg. All right. And then alignment techniques like RLHF, that's even kind of fine tuning? Yeah, mostly. Okay. And DPO, is that fine tuning too? Definitely fine tuning. Definitely fine tuning. Wow. So, okay. We got a lot to cover today, man. I think we're going to sort of do the fine tuning 101 today and tell the guys, tell the girls out there what they need to know to build with fine tuning without falling down all of these crazy rabbit holes that the entire space is riddled with. What do you say, man? Sounds awesome to me, Greg. Okay. We're going to have you back for a couple of discussions throughout today. So we're going to mix it up. And so we'll see you back pretty soon to sort of discuss some of these subtle nuances as we get into practical fine tuning. Thanks, Wiz. Can't wait. All right. Welcome, everybody. We are talking practical fine tuning today. And this is a big topic. By the end of the day today, you're going to sort of walk away with exactly how you can think about specific ways to build useful LLM applications with fine tuning. We're going to give you just enough theory to not get too confused with the space. And we're going to give you sort of that 101 level intro that is going to allow you to sort of decide what to do, decide how to do it, decide how to balance performance and cost. And we're going to mix it up a little bit with discussions throughout. So if you have questions live, please throw them in the Slido link or in the YouTube chat. We're going to have a number of opportunities for you to ask questions and to triple down on questions that I ask and I bring Wiz back on stage for. So let's get into it today, everybody. We're talking practical fine tuning. And the big goal of the day as we align our aim towards really understanding this big concept is to outline the most relevant 101 level theory concepts behind the common fine tuning techniques we just mentioned a lot in the intro we're not going to cover all of those because it wouldn't be as useful as we think it should be in a 101 level fine-tuning event we're going to talk about ways that you can think about fine-tuning and the core applications that we can use fine-tuning for and of course we'll show you how to do some fine-tuning with one of those examples so first we're going to talk fine-tuning in theory we're going to talk fine-tuning in practice we're going to talk about performance in theory. We're going to talk fine tuning in practice. We're going to talk about performance versus cost. At the end of the day, we want these things to be implemented in LLM applications that create real value for our companies and for our customers. So what's the relevant theory that we need to baseline our discussions with today? The first thing that I'll put out there is that when we think about fine-tuning, we don't want to think about it as a thousand percent equal, triple equal sign to supervised fine-tuning, but it is approximately more or less equal to fine-tuning, especially at the 101 consideration level. Why is this a useful mental model? Because we can sort of harken back to the paradigms of ML that we learned originally when we started studying the topic. We learned about supervised learning, unsupervised learning, and reinforcement learning. We've mentioned this in a recent event on DPO. The paradigms of ML can be thought of a number of different ways, and some of those ways can be kind of useful when we think about fine-tuning. For instance, supervised learning is the sort of data with labels paradigm, whereas unsupervised is the data without labels paradigm. Further, you can think of some of the ways that these paradigms overlap. Of course, there's the classification regression, the absolute classics of supervised learning. And it's worth noting here that we can use LLMs to do simple classification and regression problems. We can use very powerful LLM models to solve kind of simpler machine learning problems. And that's the first hint or key to thinking about fine tuning is that it's a natural progression from sort of the classical ml paradigm and of course if we think about neural networks we can use those for all sorts of things across paradigms which is exactly kind of where our story starts with the generative pre-trained transformer and the models, the GPT style models that have become LLMs today. This is a common representation of the way we can kind of think about what's going on when you train an LLM from scratch. And you'll notice that we're in the unsupervised paradigm to sort of leverage a lot of this data that we scrape from the internet or other sources that form the basis of the general ability model to do next word prediction. And then we can do supervised fine-tuning techniques, or also known as alignment techniques, like making it better at following instructions or making it better at specific tasks. We might need to change the structure of our model in some cases if we want it to perform a more classic task that's not next-word prediction. So in general, when we talk about supervised fine-tuning in this context, we're talking about sort of aligning it with the way we expect our models to follow instructions, or supervised fine-tuning with instructions, aka instruction tuning. And of course, RLHF, DPO, the alignment techniques sort of at the end are a continued method of fine tuning. Now, there are some details that matter here, and we're going to just talk about high level, a couple of them. So let's go back to consider how these GPTs originally sort of conceived of at OpenAI and other places, but certainly popularized by OpenAI. They started with this idea of the transformer and the GPT style transformer was a decoder only transformer. And this is sort of the type of architecture we see more and more today. With the first GPT model, it was really kind of using this decoder only stack and noticing that we can do a lot with this unsupervised pre-training. Now, to be fair, unsupervised pre-training is not wholly unsupervised. It's more of a semi-supervised approach. So we start to get into ambiguity here and we do not want to fall down this rabbit hole. So we're going to kind of leave that there. But we can kind of see that we have this supervised and unsupervised approach that we're leveraging for the original GPT. This combined with the transformer architecture, the idea of these attention layers. These are things that we've talked about in previous events that if you want more detail on, definitely check out our transformer playlist on YouTube. And this idea of using pre-trained models and then fine-tuning them was also a core idea to the GPT stack. And what we saw with the GPT model was that we could actually improve task-specific performance as we improved the underlying model. So this sort of pre-training, this unsupervised pre-training step kind of provided steady improvements with more continued pre-training. And it changed what was just fine-tuning that produced mediocre results in many cases to unsupervised pre-training plus fine-tuning results that immediately jumped up with very few fine-tuning steps to high- performance levels across things like sentiment analysis and specific nlp tasks gpt said you know what we want to work on actually improving fine tuning in addition to some other things which led to gpt2 andT-2 was like, actually, we can do some pretty cool stuff even without explicit supervision. We can do stuff zero shot. This was the introduction of the idea zero shot task transfer. And so the real interesting thing here in GPT-2 was, as they scaled up the approach, they said, well, actually, we can get really good results without any fine tuning, simply by prompting in the context window with a specific instruction, perhaps an example, which leads into GPT-3, which said, well, actually we can keep scaling this thing up and we can greatly improve performance without explicit supervision, using not just zero shot instruction, but one shot, two shot, few shot learning. And this was the moment where they were like, man, this task agnostic architecture, the general LLM combined with task-specific data for task-specific fine-tuning, maybe we can bust out of this paradigm. Maybe one day LLMs can actually just be given an instruction, just be given one or two examples, maybe even no examples, zero shot, just like humans can, and we can ask them to do things. Because humans don't require large supervised data sets and many hundreds or many thousands of examples to learn new tasks, why should LLMs? And it turns out that GPT-3 was even better without fine-tuning than GPT-2. And we see that continued today. The point we want to extract from this, and it's relevant to our discussion today, is that prompting and fine-tuning lie on a spectrum. A zero-shot prompt is simply an instruction. Translate English to French, for instance. Here's an input, give me the output. Cheese to fromage. You can also provide one example or many examples. This is sort of the idea of one shot, two shot, few shot learning. Once you have so many examples within a context window that you start running out of context window space, or it might actually just be more cost efficient, which we'll pick up that thread in just a moment, we might want to consider fine tuning versus simple prompting. Because as we go from zero shot, one shot, few shot, we get into this fine tuning space as we add more and more examples. Sometimes because we have to. Other times because it might make sense from a cost versus performance perspective. So if we can do it with zero shot, let's go ahead and do a zero shot. But if we need a lot of examples, that's when we might want to start consider fine tuning. And so this sort of gave birth to this idea of prompt engineering. It got very hot at this time, you know, as we got into this sort of language models or few-shot learners in 2020 paper. And so everybody started really focusing on providing clear and specific instructions. Context is something that has continued to be very, very important in the continued evolution of the space, but it's something that will leave discussion for another day. And the input, you know, we see a lot of one shot, two shot, few shot. We see a lot of thinking through of the examples, the sort of chain of thought reasoning. And then this idea of constraining the output format is also very, very important for some of the more complex applications that we'll build today. And when we talk about input and output in prompting, this is going to give us a key insight into how we can think about fine-tuning as well, because controlling the input and output is really what fine-tuning is all about. what fine-tuning is all about. In GPT-3's paper, they said, well, you know, recently we saw that pre-training followed by fine-tuning crushes it. But it still requires fine-tuning. And the whole idea was, can we do it without fine tuning? So this sort of space that it left for us as builders a few years later is like, well, when should we use fine tuning? And when should we use prompting exactly? When should I decide on one versus another? Rooting this back in the initial paradigms of ML, one way that you can think about this is if you can perform the task with classic NLP, you can probably also perform the task by fine-tuning an LLM. Now, do you really need a massively powerful model to do a really simple task likely not in that case you're probably over optimizing for performance and under optimizing for cost because everything is a trade-off right When we make decisions about how we want to build applications. Here's an article I found on LinkedIn that was written just a few weeks ago. It's called the AI Cost-Benefit Analysis here. And it looks at prompt engineering, fine-tuning, and pre-training. We're not going to really discuss pre-training. We kind of touched on it a little bit. I'm not exactly sure what we're supposed to get from this chart, but you can see that cost increases. And you can also see that performance increases as we go from zero shot to few shot to fine tuning to pre-training. We see this steady increase of both. So how should we think about this? Well, we can look at some numbers. Here's another recent article, if you do some Googling, that you might find, where this is looking at the cost of fine tuning an open source model like Llama 2. The total cost. And you're like, wow, those are big numbers. And we can see people draw conclusions like the cost increases by orders of magnitude once you want to fine tune the open source models. Is this precise enough as builders for us to be making decisions based on? I mean, all of these are massive numbers. Do we really need to spend that much money? This isn't the only way to fine tune, of course. It's not even the only way to fine tune open source models because the space continues to evolve. If we think about the cost of fine tuning, let's say on GPT 3.5 Turbo from OpenAI, we get this sort of per token cost structure for training, aka fine tuning, for input and for output. We also are seeing the emergence of tools like Gradient. This is from Gradient's website today. And Gradient allows you to fine tune and to host open source models through their API, similar to the way you would do it through OpenAI or Cloud or any other closed source API. And we again see this per token pricing. Very interesting. So there's a lot of stuff going on here that is kind of hard to square exactly. When do I need fine tuning? Well, before we open this up for discussion with Wiz, let's just think about how when we move from prompt engineering to fine-tuning, we're kind of optimizing the LLM's behavior against a specific task. That's what we want it to do. We want to constrain its behavior from the general everything to something more task specific. And there's a number of different ways to think about this idea of task, but structurally what we're doing when we move from prompting to fine tuning is removing examples, one shot, two shot, few shot into the actual training of the model. So we're rewiring the neurons and the weights. We're recalculating the weights, the parameters within the model so that inference costs the same amount, but I don't have to put as many tokens in on the front end. I don't have to put as many tokens in on the front end. And this has implications for how we would make a decision based on cost. So we're kind of focused in on how the model should act, how the model should behave. And with this, I want to sort of bring Wiz back up onto the stage to have our first little discussion here and if you guys have questions go ahead and uh shout them out in the chat but i wanna i wanna sort of ask like there's all this stuff out there there's all these articles on cost i'm not exactly sure what to make of them um when you think about the trade-offs between using simple prompting and fine-tuning how are you thinking about this wiz I mean, one of the first things I think about is this idea that, you know, there's the per unit call cost or however you want to define it. And then there's the actual cost of fine tuning, right? So prompt engineering is awesome. It is something that we can kind of include without every prompt. It's very flexible, but it does mean we're adding some fixed amount of context to our context window per call, which results in greater costs, both if you're hosting the kind of inference on your own, on your own, or if you're using the, you know, API is like open AI, etc. So there's this idea that, you know, there's these two kinds of costs that we want to keep in, in mind, when we're, when we're thinking about our, our costs for fine tuning versus prompt engineering. Okay. And then this kind of cost of 7B versus 13B versus 65B article that is sort of top ranked on Google recently. I'm just doing a little bit of Googling here. What are the costs of fine tuning? Is that the right way to think about it? I mean, aren't there some techniques out there that people have heard of like PEFT and LoRa that can maybe help reduce this cost a little bit? Like what's going on there? I mean, should I just sort of trust these big numbers at $1,300 to fine tune a 7-beat? Does that sound right to you? It depends and it varies so much that I don't think that there's going to be a single source or single resource that you can look to that will really actually tell you kind of, you know, generally the cost. You can use it to understand, say, relative cost, right? So bigger models will cost more. How much more is going to depend very heavily on the kinds of fine tuning that you use. And keep in mind, right, heft parameter efficient fine tuning is all about methods that allow us to fine tune models without worrying about as many parameters. So when we scale into these kind of very large language models, right, sort 70 to 100 plus billion parameter models, I think it's important to realize that we do have ways to quote unquote cheat out, you know, additional cost savings by using parameter efficient fine tuning methods. So it's hard to find a single source of truth in terms of cost, though you can generally still assume big model will cost more to fine tune than small model just due to the nature of the fact that we need. Even if we use parameter efficient fine tuning, we're probably going to be training more parameters with a larger model. So even if we use parameter efficient fine tuning, we're probably going to be training more parameters. So this idea of training all the parameters versus training just some of the parameters, this is a key distinction too, for people to keep in mind, isn't it? Absolutely. Yes. Without a shadow of a doubt. It is so important to understand that none of this training is going to happen in a vacuum. And because it's not going to happen in a vacuum, these kind of vacuum charts aren't really going to help you decide what costs you're going to incur. And that's going to come down to doing some research and some experimentation to figure out, you know, what's the best strategy. But almost never are you going to want to train all the parameters in your model uh in this day yeah yeah okay and like we we can't be sure like when open ai is fine tuning for us whether they're doing some sort of parameter efficient fine tuning technique or a full fine tuning. And we can't be sure if it's behind an API in general, if people are doing parameter efficient or full fine tuning. But do you have any guidance on how to try to benchmark and baseline performance as you're doing fine tuning for people that are starting to dip their toes in? Like, is there an amount of money that you think you should be spending on an initial test to see if fine tuning does the thing? You know, I know we do a lot of stuff in class and it's relatively cost effective. What insights can you provide to the audience that that might help them out if they want to dabble a little bit in fine tuning yeah I mean if you're going to be fine tuning like a production use case you're probably going to be dropping like a thousand bucks on the fine tuning process uh you know I I can see situations where it might be less than that but uh ultimately uh you're going to hit that 1K pretty easily, especially if you're using kind of these larger models, you know, 13B plus. And that just comes down to the fact that we need GPUs to run these things and they have to run for a while. And if we want to do like a real kind of, you know, full quote unquote, have fine tune, we need a lot of samples, right? So you might be able to get away with like 500, but you might need up to 50,000. And so to ballpark it, I say you're definitely going to be dropping 1K to get the ball started rolling. Yeah. Got to pay to play. Okay. We got a couple of questions and I think they're going to be better suited for the next discussion sesh so for now let's go ahead and keep rocking it here thanks wiz we'll have you back up in just a little bit um shout out to chris walker we are going to get back around to that one here in just a moment um if you want to know more about peft and laura check out our efficient fine tuning with peft la Laura event that we did recently. Hopefully that will help you out. As we think about building prototypes with fine tuning, we're generally doing this thing that we're pretty comfortable with. We're taking a pre-trained LLM off the shelf, and then we're sort of adapting it, right? We're sort of specifying it. We're taking this general thing. It can do lots of stuff. And then we're either specifying it to a specific behavior or task or a specific domain. Now, we've talked recently in our alignment series about how after you do this, you might as well go ahead and align, whether it's with RLHF, RLAIF, or might as well go ahead and just pick up DPO. It's becoming the industry standard. Alignment is sort of a simple thing to do after. And when we are trying to adapt it to a particular behavior or task, we're trying to sort of make it more helpful at that thing. When we align it, we're trying to make it a little bit more harmless at working with us to produce that thing. And both of these are useful and there's further alignment aspects that we'll get into actually next week in our REFT event. When we sort of zoom in on this adapting to a particular behavior, task, or domain, we can kind of break up three primary forms of fine-tuning. Now, this is not something you'll find anywhere. This is sort of our conceptualization, and we'd love as much feedback as anybody can give us on your interpretation and how useful you find this. But there's kind of the classic, the OG, training it for a particular task, right? This is not new. This goes back all the way to classic NLP and supervised learning where we're training the behavior of the LLM response. We get some sort of specific thing that we're doing. Summarization, right? Sentiment analysis. These are some of the absolute classics. Choosing which one, you know, which multiple choice option comes next in the sentence, if it's not relevant to the first part. These kind of sort of classic benchmarks or tasks. Then there's sort of this interesting space that's opened up where we can talk about constraining the input-output schema. This is less about the behavior and more about the format of the response. So for instance, you instance, we got the question in the chat from Chris, can we increase function calling performance? When you call a function, you want the format specifically to be in JSON. We can constrain the format of the output to be something that can be used as an input to something else. Right. And that's sort of the idea of function calling. There's a lot of ways we might want to constrain the input output schema and task training and constraining the input output schema are not mutually exclusive. They are not independent of one another. not mutually exclusive. They are not independent of one another. Nor is the third piece of this, where we can sort of train new language. Now, you could attempt to train a completely new language, like teach Chinese to an English-speaking model. You're probably not going to get nearly as good of results as you would if you did some unsupervised pre-training on the native language. More potentially usefully here, we're going to be using the language training piece to train the LLM's ability to interpret new words. So it's not necessarily that they were trained to understand them from scratch as an unsupervised pre-training modality, but we're sort of training the ability for them to look at input-output pairs and then draw pair, draw distinctions between words and connections and patterns between new words that are rooted in the patterns between words that it was trained on. So this is very useful in particular domains. And this is something that we'll see today during our build. This sort of all leads back to this idea that everybody's talking about today, which is smaller language models. They're like they're more efficient, they're more accurate, right? Cost and performance. You got both. And they're easier to sort of disambiguate. They're easier to sort of say, okay, how'd you get that answer exactly again? And potentially they're also easier to secure due to their smaller size and their ability to be aligned with the exact data that's used to train them. Now you can train a small language model from scratch using an unsupervised pre-training approach but you can also do fine-tuning and fine-tune something down to a smaller version. I want to sort of, before we open up this next discussion, I want to, I want to look through a couple of examples for the primary forms of fine-tuning. Task training, constraining I-O schema, and language training. And let's think about this in the, in the sort of terms of like, how you're going to like, how you're going to like, of like how you're going to like, how you're going to like if I something, okay, check this out. So if we wanted to like JSONify, right. The, again, the output format, again, we could, we could tell in a prompt, Hey, always output JSON. It's going to work. It's going to work pretty well sometimes. It's not going to work all the time. You might need fine tuning. You could also constrain an LLM to produce Docker output formats or YAML output formats, right? You'll notice that a lot of people, a lot of big R&D labs are coming out with code-specific models. Probably they're going to leverage a lot of this for their internal code generation, right? Boilerplate code, sort of a thing of the past, or at least it's rapidly becoming one. So we can sort of iffy in programming, right? We can also sort of iffy in marketing, right? Maybe we want to turn some product description, some interesting thing about a use case. We want to turn it into a tweet. We want to turn it into a direct mail. We want to turn it into a LinkedIn post. Now, in this case, we're constraining the I.O. schema. Yes, maybe we have a product coming in. Maybe we have some other sort of input. But then the output is going to be a tweet or a direct mail or a LinkedIn message. And this is something that you could fine tune. Or if we wanted to iffy across domains, right, we could sort of turn regular language into legalese. Not sure why we would want to do that. regular language into legalese. Not sure why we would want to do that. Or we could turn regular language into sort of highly intelligent doctor speak, right? Extra big words or researcher speak. And this is something that we can very much do today with fine tuning. So I want to kind of bring, bring whiz back up now. And we're going to have a little bit of a, another discussion here. It's all about performance versus cost. It's all about performance versus cost. You know, here's a, here's a question we got from a learner in class recently, who is in a large enterprise. And he says, if I'm already happy with the performance of my LLM application, but I just need to reduce inference latency, I just want to make this baby faster. Should I consider fine tuning? Yes. Yes. Well, how is that so clear to you? The kind of way that I would look at it, less so from like the specific latency point of view, but more so from the way that we're gonna utilize our hardware, right? Is if we don't have as much kind of glutting up our context window, right? So if we have like these prompts that have six to seven different examples, we can more efficiently use our hardware. We can make it so that we have more, you know, resource to handle multiple requests at a time, which is gonna lead to a decrease in inference latency. The idea again, is that especially if we're talking about a kind of per request cost, right? We're gonna be able to be more efficient with those requests if we've fine tuned all of that kind of extra stuff out of our prompt, right? And we've baked it into the model. I think that's the way that I would think about it. So you're saying that by moving the prompt, the required prompting tokens out of the prompt, you're fundamentally gonna speed up the inference. Yes, you'll wind up with reduced inference latency over many calls. That's true. Okay, okay. Yeah, that's pretty cool. So yeah, I mean, this know, here, this is just a common example of a question that we'll get that, you know, really isn't necessarily clear to people what they should be doing. And I've got another example I want to talk about here on the cost versus performance side as well, which is, you know, is it better to pick up a large domain specific model? For instance, you know, the medical domain has models like MedPalm and MedPalm 2 from Google, or is it better to sort of pick up a Mistral 7B and then teach it medical stuff through fine tuning? Like, is there an answer to this? I, there's everyone's favorite answer, which is that depends. And of course, I mean, this one's a lot easier, though, right. So what it depends on is two things, which is the, you know, how much are you saving by using a smaller model, right? So these larger domain specific models are fantastic. I mean, you can't go without saying that they're incredible, but they're very expensive to run, host, interact with, through APIs, all of this, right? Now, is it the case that these smaller models will be worse than those larger models? I mean, probably, yes yes right um but that's the second piece of information we need to consider how much worse you know it's the case that if through fine-tuning we can get to uh you know 90 80 uh on whatever our desired metrics are and we're paying 80 90 less that's a that's a trade-off that is well worth considering in most situations. And to be clear, for Ratna, we're talking about an off-the-shelf model, so not pre-training. If this question were pertaining to pre-training, the answer would be not even close. You know, pre-training is so expensive. But an off-the-shelf large model is going to potentially cost you more for very little improvement or very little enhancement to your end user's experience, which is ultimately what we care about. Yeah, yeah, yeah, yeah. Yeah, and it seems like the evolution of this space is such that the domain specific models, they're more and more coming onto the scene and, you know, they're sort of coming out also, you know, we're seeing this with the code models in particular in different sizes. So as we see sort of different sized domain specific models, this is going to make actually the answering this question even more particularly nuanced. Is that right? Yes. Just as we went through with BERT, if you can remember all the way back to the BERT days, right? When we went from kind of these big general BERTs to smaller BERTs, distilled BERTs, Roberta's, right? Then then cybert then all of the as we go through this evolution where we start building you know a hammer when we need a hammer and a you know and a screwdriver when we need a screwdriver obviously uh it's going to become a little bit more of a discussion but right now uh it's kind of like we're building the world's biggest hammer Now it's kind of like we're building the world's biggest hammer that's definitely going to get that nail in every time versus like a 100X smaller hammer that gets a nail in 99% of the time. Right? So this is the trade-off right now. Okay. Okay. And the last thing I just want to sort of talk about before we go into the build today, and if you have questions, we'll have approximately 10 minutes or so to discuss additional questions you guys have, is sort of going back to these three primary forms that we're offering up as sort of ways for people to think about fine-tuning. We've got task training. We've got constraining the IOS schema. We've got language training. We've got task training, we've got constraining the IOS schema, we've got language training. We want to remind people that it's not always just one or the other, right? A lot of times you can curate the data so that you're doing a specific task within a specific language domain. And even that you're also constraining the IO schema, right, it's like you could be doing all three at once. It doesn't have to be one or the other. How would you recommend people to think about, you know, what they should be doing when they pick up a model and try to fine tune it? Yeah, I mean, I think the wisdom of get as far as you can with prompt engineering, and when that starts to fail, fine tune, or if your prompts are astronomical in size, right? Like the context you're providing to your LLM through your prompt is like three times or four times what the actual user's queries are, right? That's when I would broach fine-tuning. When it comes to the data, though, we just need to show it examples, right? Show it examples of those behaviors. Show it examples of the language that we care about, how that language is used, what that language means, right? Those are the things that i would focus on and it always comes down to uh we're gonna need to use some combination of things some combination of of uh you know these processes that's what we're gonna see today right we're gonna have fine tuning uh but we're also going to continue to use prompt engineering because it's super useful right and it lets us do uh really flexible cool amazing things uh and so this this you know fine tuning or prompt engineering the answer is yes meme is uh i think still relevant yeah yeah yeah yeah so so we're ultimately specifying a more general capability model i, that's what we're always doing. We might not be moving towards task-specific in a super hyper definition of the word task, but we are moving towards something that's more specific and more aligned towards our particular use case. Yeah, that's right. Yeah. That is right. That is right. Okay. So you see these words are slippery. Thanks, Wiz. We'll have you back in just a minute to run through the demo. So, you know, this is the difficulty is the language surrounding this stuff is challenging and getting a handle on it, as the field evolves is also challenging. For today, what we're going to do is we're going to walk you through a fine tuning example. And we're going to use a Mistral 7B, Instruct V 0.2, a great off the shelf model, it's already trained to be very helpful across many tasks. But what we're going to do is we're going to make legalese easy. So we're going to take crazy legalese speak, turn it into plain English, and we're going to call it legal easy. And we're going to have Wiz walk us through how to fine tune MISRA 7B for legalese. Now, remember to throw those questions into the slide up or the YouTube live chat to keep the discussion rolling. Wiz, over to you, man. Thank you, Greg. Yes. Okay. So let's talk about legal easy, right? The idea is simple. We have, we interact with in a day-to-day sense, a of really dense legal uh context i mean especially when we're entering the fine fine tuning space right we're we're thinking about model licenses we're thinking about you know when can we use these things acceptable use case all this other stuff right so we're always interacting with legalese and uh it's very dense it's hard to understand unless you have a lot of background in that domain. And so we're going to go ahead and we're going to use our fine tuning today to produce basically a summarization tool, which is going to natural language and very, very terse natural languages, we'll see. So it starts, as it always does, with grabbing some dependencies. We're just going to install some dependencies. Easy peasy. Away we go. The next thing we're going to do is grab some data. The data we're going to use is from this paper, plain English summarizations of contracts. And you can see that it's in this legal summarization repo, which has a JSON, which has all of the information that we need, right? Perfect. The original text is what we care about. This is just a blob of legal text, like a terms of service agreement. And then we have a reference summary, which is a short natural language summary of the legal text. That's it. When it comes to our data acquisition, we're just going to clone the repo. And then we're going to load that tldrlegalv1.json file, which is going to have our examples that we're going to use today. which is going to have our examples that we're going to use today. We'll just load this into a list of JSON objects. And then we will basically all we'll do is we'll add those to our data set from that list. Easy. Now we have a data set. And that data set is going to be 85 objects, which have this original text and reference summary features. We can move from there to checking it out, right? So we can see the original text has text like, welcome to the Pokemon Go video game services, which are accessible via the Niantic Inc, Niantic mobile device application, the app, to make these Pokemon Go terms of service the easiest, blah, blah, blah, blah, blah. And it comes out as the summary is hi right so i mean that's a that's a pretty good summary this is just saying hello uh what we want to do though is we want to take this from this format which is hard to work with a little bit cumbersome and move it towards this instruction format as is desired by the mistral model right so we So we'll see in a second the Mistral model card expresses how it prefers instructions to be delivered to it, and that's it. Now we're going to look at the instructions prompt, right? All we're going to do is we're going to add these tokens for legal doc and end legal doc. We're going to supply our legal doc in between those tokens, And then we're going to end our instruction. And then the output will be this summary, which says hi in this case, and our end sequence, end of sequence token. So we'll create a formattable string that we'll use, we'll fill in the blanks, let's say, right? No worries there. Then we're going to move on to creating a helper function that will wrap any of our rows in this prompt format, and it will, you know, conditionally return the response. So for generation purposes, we don't want to include the natural language summary or the stop token. So we're going to omit those from this process. Then we can see that it works and we see that it works. That's great. And let's just, you know, before we go any further, let's like look at what this will do right in actual Mistral. And you can see here, this is the instruction format Mistral cares about. We can see, you know, please convert. This is our instruction. And then the output. Very long. You know, very, it's not really a summary. And it's certainly not supernatural language, right? So let's go back to our notebook and see what we can get it to do. We're going to split our data into train and test as we always do. And we'll have 76 rows in train, nine in test. Perfect. Now we're going to load our model. This can all be done on the T4 instance. I'm showing it in the A100 instance just because, you know, it's a little bit faster. But you can do this in the T4 instance. The maximum GPU RAM that was utilized was 15 gigabytes or 15.8 gigabytes to be clear. So it will fit in the T4 instance. But you're welcome to use a V100 or A100 instance if you have access to those. We're going to use some technology that we've talked about in other events. I'm not going to go into it now. But the basic idea is we're going to load this in four bits and then we're going to map it to our GPU. Easy peasy. We're going to set up our tokenizer so that we don't get as many complaints when we're doing the rest of the process. And we're going to set up our PEFT config. This is our LoRa config. So this is going to be how we tell our TRL trainer, which is what we're going to use to actually train the model or fine tune the model, what kind of LoRa configuration to use. So we're going to use 32 as our rank, 64 as our alpha, and then we're going to target all linear layers with no bias and the task is causal LM. Quick rule of thumb here, alpha is usually just twice that of rank. That's it. Then it comes to fine tuning. In fine tuning, we have, we've kind of, you know, got to the end of the line. We've set up our data, we've loaded our model, and now it's time to actually train. I have some text here that's going to help understand some of the parameters we're using so that you can go through it at your own time. But for right now, all that matters is that we set these up, right? These are hyperparameters. These settings are used to ensure that we're keeping things within the memory limits of the T4 instance. We're going to train for four epochs, right? Which is not very many. We're going to use the fused optimizer. We're going to keep everything in FB16 so that we don't have architecture issues. You know, BF16 requires the Ampere technology, which is our architecture, which is not on the T4 instance. So this is going to make it so it's still possible to use T4. And then a bunch of these parameters just come from the QLORA paper. And that's why we're using them. We'll have this optional step where you can use your Huggy Face write token to automatically push your model to the hub as you train it, which is useful for basically model versioning. Then we're going to go to our SFT trainer from the TRL library. This is our supervised fine-tuning trainer. We're going to load it up. We're going to pass in our model, the args we just set above, point it to the training data set, point it to that helper function that creates those prompts, point it to our PEFT config to tell it how to build the LoRa layers. And then we're going to add these a few flags the flags are basically saying we want to use packing to make our training more efficient and then we don't care to add special tokens because we've already added them in in our prompt creation we let this thing train it trains for four epochs you love to see it we save the model out we free up some memory so we can run it in a pipeline. We're going to merge the model just due to the Colab versioning of Transformers. We can't use a auto-pef model for causal LM with a Transformers pipeline object, so we're going to merge and unload our model, and then we're going to load it into a pipeline, and then we're going to see it on one of the test examples so this test example is basically a big long block of legal text that don't care to read we're going to pass it into our pipeline to generate from our new model our new model is going to say don't distribute our game pretty good right that's a good summary uh and the references you may not give copies of the game to anyone else or try to make money from anything we've made, which is in essence just saying don't distribute our game, which is nice. We'll look at another example. The next example is, again, a lot of stuff I don't care to read with words like jurisdiction, applicable law regulation, all very fun words, but not useful words. And this generated a summary in natural languages, don't use the SDK for illegal stuff, right? And then the ground truth is stay within the law and license agreement, which is basically the same thing, right? So this is the idea that we are able to fine tune our model to take a complex input in full of legal jargon and output a natural language description. That's very short to help us get things done. And that's it. With that, we will pass you back to Greg before we take some questions. You're muted, Greg. Awesome, thanks, Chris. You're really great. Awesome. Thanks, Chris. And that is how you do fine tuning. And of course, the how is often a little bit easier these days than figuring out the why and what. Now, if you wanted to get more details on quantization and QLORA, then go ahead and check out our event that we did on this, where Chris walks through in more detail some of the aspects that we kind of glossed over in the demo run through today. And in conclusion, where we kind of get to by the end of this is, you can kind of think about fine tuning as approximately equal to supervised fine tuning. fine-tuning as approximately equal to supervised fine-tuning. This isn't a perfect analogy that holds up at every level as we get more and more into the nuances, but it is something that's a useful way to think about it at a sort of one-on-one level. We can think about this task-specific spectrum of prompting and fine-tuning as we're building applications to always start with prompting. And then three types of fine-tuning that we offer to think about as useful ways to conceptualize what it is you're doing, task training, constraining the I-O schema, and language training. These are not independent, they're not mutually exclusive, but they are ways that might be useful as you think about problems you're trying to solve for things that you care about or in your company. And with that, we're going to go ahead and get into our final Q&A portion today. Wiz, welcome back up to the stage. We got some nice questions coming in. I'm going to just rapid fire a couple ones at the bottom of the list here related to the actual code. What is Atom Torch fused and is that target modules equals all linear new? And you're muted now. I'm muted, sorry. It does this thing. Okay, so Atom Torch Fused is just Atom W, but it's fused. All this means is that it's using specific CUDA operations to perform the operation more efficiently. So basically, instead of doing like a loop, it's going to use a more efficient way to process through the optimizer updates. Okay. And then is the target underscore modules equals all linear new? I believe it is. Yes. Okay. Astute question. And then a big question here at the top here, how much data, time and money do you think it would take to set up a domain specific model? How complex do you think this is? If you're talking about pre-training, a lot of all of those things. You know, it's you want into trillions of tokens. You want probably, you know, a 7B model. Maybe you go down to 1B because you want to be cool like Fi. But you're still going to spend a lot of time, data and money. The complexity is maybe relatively low compared to what it used to be, thanks to a lot of really fantastic innovations and tooling that's available to us today. But the rest of that stuff, data, time, and money, high set of domain-specific model. specific model. Yeah, I'm reminded of the event that we did on Smart RAG with the guys at RC that said they could sort of fine tune a larger system end to end using about a million documents, something like that. So yeah, lots and lots and lots, I think is the answer. And that's why, you know, a lot of times for our particular domains, as you see domain specific models come out, pay attention and be one of the first to pick up and leverage those. And I think that that'll be something that proves very, very useful for the companies that do it early. for the companies that do it early. And then how was the legal text formed from this PDF? Exactly. I'm not sure that that question. I don't quite understand the question. We used a JSON blob as our data set today, which basically just contains fields that have text, and that text is related to either the legal blob or the natural language summarization of the legal blob. Okay. Yep. Pretty straightforward. I want to just end on this question from Rami that we had a little bit earlier in the YouTube chat. Should we fine tune using pure pre-trained models or can we use a model that has already been fine tuned? In other words, stack fine tuning to enhance the results. This is a great question, Rami. You should always use the model that is closest to your desired behavior on your task every time. Whether or not this is a already fine-tuned model or a base model or a instruction model or a, you know, it goes on and on, doesn't matter. The idea is we want to start as close to our objective as possible and then move the rest of the way. So if a model exists that's already pretty good at what you want to do, you should assume it will be easier to make it even better at what you want it to do than a model that's like somewhere in outer space, right? And this is the whole idea of these domain-specific models, you know, or adding instruction or tasks to instruction tuned models, right? We don't want to use like a law model to create a medical model, for instance, right? We want to kind of use a regular based model. If we already have an instruct tuned version of our model, adding new instructions to it is easier than re-instruct tuning it altogether. So that's how I would approach that. Yeah. And I'm reminded of sort of the rule of thumb, like just grab an instruction tuned version off the shelf. I'm not aware of any applications that cannot deal with an instruction-tuned model and must have the base model before you start to try to get them to do your task that is ultimately going to help humans. So I mean, I think that's another rule of thumb that you can keep in mind, Rami, is just grab the instruction model just like we did today. If there's a v0.2 that is better than the v0.1, grab that one. That was a place to start. All right. Thanks, Wiz. It's time to go ahead and wrap up. Thank you, everybody, for joining us today. If you found this useful, please tell a friend, colleague, or manager about specifically what you learned and how it's likely valuable for things you're building in your context. Join us next week to talk about how to align LLMs towards a little something different, towards doing math better with REFT. And of course, like, sub, and if you've already liked and sub, please ring that bell to stay up with updates as we drop new stuff on YouTube every week. If you are seriously ready to accelerate your LLM application building from prototype to production, our next cohort of the AI Engineering Bootcamp starts on April 2nd, submit your application today. It is a tough course and it is one that is a lot of work, but it's very meaningful on the other side for a lot of folks that are taking it. If you enjoyed hanging with us today and you're not in Discord yet, jump in Discord, throw an intro down, start meeting some people, start joining community sessions, and you can start building today with all of the concepts and code that we've shared on YouTube directly through our awesome AIM Index. Or you can start learning directly through our open source courses like our LLM Ops, LLMs in in production cohort one recently open sourced course more to come on open source courses in the future so stay tuned finally if you have any feedback please share it with us in the chat or in the feedback form that you receive right now or on luma as always keep building shipping and sharing. And until next time, we will most certainly keep doing the same. See you all next week. See you next time.
Practical Fine-Tuning of LLMs
GPT-4 Summary: Unravel the complexities of fine-tuning in LLM applications at our enlightening event, designed for everyone from learners to AI engineering leaders. Discover the nuanced world of Supervised Fine-Tuning (SFT) and its pivotal role in building, shipping, and sharing effective LLM applications. This session aims to demystify the primary forms of fine-tuning crucial for enhancing user experiences, including task training, constraining the I-O schema, and language training. Dive into enterprise use cases, live fine-tuning demonstrations with the latest tooling, and a comprehensive Q&A session addressing common theoretical inquiries such as the distinctions between PEFT, LoRA, distillation, and the tradeoffs with prompt engineering. Whether you're seeking to understand the practical applications of fine-tuning, conceptualize SFT to peers, or develop a strategic approach to when and how to fine-tune your LLM applications, this event promises invaluable insights at the frontier of AI development. Join us to elevate your knowledge and application of fine-tuning in the ever-evolving landscape of large language models.

Event page: https://lu.ma/practicalft

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/jxjJcjqP6ZwEhop9KQjwzC

Speakers: 
Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/AIEbootcamp

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/AFwLq948NNm6dh3Z6