 Welcome back to session eight. This is on embedding, fine tuning, and we're going to go ahead and see how we can do this in a tool like Lama Index. Now, this is bringing a couple of things together. We want to align ourselves to everything that we're doing here. We're going to do a quick review, review Lama Index, and then we're doing here. We're going to do a quick review, review Llama Index, and then we're going to build a veterinary camelid index. We're going to build a llama index with Llama Index. All right, you guys ready for this? Then we're going to fine tune it because vets say some crazy words. Remember, why rag? Well, because LLMs lie to us, because we need to fact check them, because we need references, and we need to sprinkle some references into our prompts. But also remember that we talked about how important it is when you have very specialized domains with very specialized language that you may want to consider that as soon as you build up a RAG system or in the process of building up your first RAG system, you consider fine-tuning embedding models. That's what we're going to do here today. Because the language is so specialized, it's just not anything anybody would ever say to anyone and you would randomly find on the internet. So let's take a look here. During this RAG system, which of course looks like this, it combines dense vector retrieval and in-context learning. We're going to leverage Lama Index to build this thing. And we're also going to use Lama Index to fine-tune our embedding model. So recall, Lama Index is a data framework. It's all about that data. Lama Index uses nodes as NLP documents and PDF documents, aka documents as source documents. Those first-class citizens are just chunks of source docs. They're called nodes. The parsers allow us to take the docs and then chunk the docs. They're called nodes. The parsers allow us to take the docs and then chunk the docs and create those nodes. Okay. Query engines is what it's all about. This is the big idea with Lama Index. So what is a camelid, you might say? Well, camelids include camels, of course, llamas, alpacas, vicunas, guanacos. Hey, look at that. If you were wondering where any of those names came from, they came from the camelid family. And if you're wondering why we don't have any more camelids, well, there's none left in the picture. So maybe that has something to do with it. We've moved on to winds like Mistral and Zephyr. But for this, we're going to look and dig super deep on camelids. Shout out to Ohio State for having really in-depth vet info in their research library on camelids. Apparently, this is where you'll find the International Camelid Institute. And this kind of place, if you're doing work in a place like this, this is the kind of place where you might consider fine-tuning your LLM embeddings, because that's probably definitely going to help improve some of the retrieval and some of the generation. Because otherwise, if you just don't understand the words, you're just going to have a tough time. So building this camelid index, this llama index with llama index looks similar to other indexes that we've built with llama index. And if you consider the ways that you might go about improving retrieval, Lama Index is constantly building out these capabilities. But they're often talking about a lot of different ways that you might start to do more interesting and complicated things. And one of those ways is fine tuning of embeddings. In this particular case, because we have such specialized language, we're going to fine tune those embeddings. The way that we fine tune embeddings is we're going to, and if we're going to have another like joker in here, we're going to have to kick them out again. So bring it. Please mute if you are messing around. The ingredients for fine tuning embeddings are these question retrieved context pairs, right? So what we're going to do is we're actually going to create these question retrieved context pairs, and then we're going to take an existing embedding model, and we're going to train it, so to speak, on CAMELID research paper context. That's it. And we use sort of a very simple approach here using a built-in Hugging Face sentence transformers loss function. And it's really not that complicated. What we see when we do this is that our hit rate, our ability to find the appropriate context, given any particular query, actually improves. And so if you have very specialized languages, you might consider fine-tuning embeddings. And the way you do that, we're going to show you right now. right now. Chris, Camelid Embeddings. Oh yeah, let's get rocking with some Camelids. Okay, hopefully you can see my screen. The basic idea here is we are going to fine-tune our embeddings. So why would we want to fine-tune our embeddings? Well, as Greg said, you know, especially in these kind of veterinary papers, there's just so much language that we have, you know, no idea about or we don't know how it's related to other, you know, tokens in our in our corpus or, you know, it might have one meeting and kind of common parlance, but have a totally different uh in in the case of this specific application so the thing we need to do is we need to I'll link the collab sure one second sorry here we go thing we need to do is fine-tune those embeddings right so first of all get a bunch of dependencies second of all we're going to grab our OpenAI key. Then we're going to get the camel data from our data repository. We're going to go to high-performance RAG, and we're going to download camel papers test and camel papers train. You can see there's a bunch of crazy papers about camelids and and uh you know that's great what is the intuition between the behind the cue retrieved answer pair idea to fine-tune the beddings is the last binary type of last so the the way that they do it actually is they they make the assumption that every other context in the QA pair data set is a counter example to the found context or to like the selected context. And then I can't remember the specific loss function, but I'll bring it up and I'll show you guys. Now that we've got just a lot of papers about camels or camelids to be more precise uh we're going to go ahead and load those uh we're going to load those using our simple directory reader which reads directories our simple node parser and our metadata node our simple node parser is going to uh parse out all of our uh documents into nodes for us yeah yeah i'll bring up the loss uh function for sure once we have these two corpuses we're good to go now we're going to generate QA embedding pairs which we're going to do with everyone's favorite of course AI so we're going to use open AIs gpt 3.5 Turbo to generate QA embedding pairs. Then we're going to save those as a data set. And then we're going to do the same thing for our validation training set. So now we have our validation and we have our train. Everything's good so far. Next, we're going to use the sentence transformers implementation to get BGE small 1.5. It's just a good embeddings model. It's trained on a big corpus, and it performs very well on our task, which is the retrieval task. So that's why we're picking it. The embeddings leaderboards update very frequently, so you can use whichever one you want that performs the best at whatever tasks you need to do. Now we're going to use the sentence transformers fine-tune engine from Lama Index. Thanks, Lama Index. We pass in our training data set, the model we wish to fine-tune, the output that we wish to have our model be saved in, our validation data set, and then the number of epochs we're going to train for. Of course, we could train for more or less time. It's totally up to you. But the idea here is that we have, you know, the same kind of training process that we would for a normal model, but this is for a sentence transformers model. And it's to kind of drag, right? We have these embeddings. If we just imagine them in 3D space, right we we know they're kind of in this cloud and their proximity to each other or their direction from the from the the origin uh you know is it in a particular spot and we're just kind of dragging them around or moving them around re reclustering them in that space in order to align with our actual corpus of documents better. So that's a way you could visualize this if you were a person who liked to visualize things. Once we do all that preparation, we do everyone's favorite step. We call.fintune, and we see it go. And then we can grab our fine-tuned model out of our fine-tune engine now we can set it up as its own embedding model and what we're going to do now is evaluate that embedding model so we've created it and that's good but we need to evaluate it we're going to evaluate it with this. So there's a lot of metrics that you're going to get from this evaluation. We're only going to really care about the map at K. So this is the mean average precision at K. I believe it's map at five that it reports back to us. The idea here is we just want to make sure we're retrieving the right kinds of documents in the top five documents retrieved, right? So how often are we doing that, you know, is kind of what we're caring about here. So we want to, for the most part, always retrieve the correct document in the top five. Now, obviously, we're not going to get to perfect with two epochs of training on a very strange corpus, but we can see that with this evaluation, which is all done through wonderful abstractions, thanks to sentence transformers in this case, not Lama index, We can see that our base unfine-tuned embedding model receives a MAPIT 5 of 0.76, and our fine-tuned embedding model receives a MAPIT 5 of 0.79. So we do see that there is a real increase between the two. Again, this is two epochs on a very, very strange data set. Ideally, we train for longer in order to get this result even better. But it just goes to show you that even with the smallest amount of effort, we can improve these systems to be better at the tasks we need them to perform. One thing I do want to point out or mention, tasks we need them to perform one thing I do want to point out or mention when you're looking at your map at K scores it is important that you set your retrieval K to be the same value as you see uh in the in the metrics right if we have a very high map at K or map at five but we only retrieve three documents we're potentially shooting ourselves in the foot. So you want to make sure that you align the desired behavior of your retrieval pipeline with the metric that you're looking at. Just to point out the fact that, you know, you might not see, let's say we did RAGUS on this, but we kept it at only three retrieved documents, we might not see an improvement. And that's because we weren't actually looking at the right metric in order to make a decision about which is quote unquote better at that task. But with that, I will kick it on back to Greg. All right. So we saw those numbies going up. That was pretty cool. We didn't even train that long, but it did help. And that's pretty legit. You know, that's kind of the big idea. There it is. In a nutshell, fine-tuning embeddings. Lots of other things we could do for any given RAG system. All sorts of fun retrieval. All sorts of fun different node parser stuff in Lama Index to play with. All sorts of different evaluation things we could potentially do to instrument this thing and measure different numbers, see if they go up too. But that's a wrap for this little sesh. There are many ways to enhance retrieval and thus generation. Fine-tuning is one that you might want to pick up for very specialized domain language. And, you know, an example of that is the VET LLAMA index with LLAMA index. So as we wrap up session eight, the final session, session nine, that's not directly related to you guys presenting what you've got today, is going to be on deployment of your app. Once you've got all the logic, all the brains, all the everything in the RAG system, it's time to serve that thing up to users. And so we're going to see how to wrap everything in a chainlet front end, deploy it to Hugging Face, and make sure that you've got that killer demo that you can show live by the end of the day, night, or morning, depending on where you are in the world, as we start to make it into the final hours of the first annual Chativersary Hackathon.
Session 8: Fine-Tuning Embedding Models for RAG Systems
What you'll learn this session:
- How to tune open-source embedding models to align with specialized language, like that used for research

Speakers: 
Dr. Greg Loughnane, Founder & CEO AI Makerspace.
https://www.linkedin.com/in/greglough...

Chris Alexiuk, CTO AI Makerspace.
https://www.linkedin.com/in/csalexiuk/

Apply for one of our AI Engineering Courses today! 
https://www.aimakerspace.io/cohorts