 Welcome, everybody, to part two of the Chatterversary Hackathon, and this is going to be the deep dive overview of RAG. So let's jump right into it. We're going to go ahead and align our aim here. We want to make sure that we are understanding each and every one of the core components of RAG. We like to break down RAG into two primary pieces, dense vector retrieval and in-context learning. We're going to think of RAG throughout the day as question answering over documents, and we will build a RAG system in the next portion of today's event. Now, it's important to think about why RAG. This is from Deep Learning AI's The Batch this week. And it's amazing that just a year after ChatGPT, the number of closed and open source models that we have to learn from, to leverage, to pick up and use in our work. From OpenAI to Microsoft to Google to Anthropic inflection perplexity there's all sorts of models out there today and the apis that we can use to pick up models off the shelf AWS all the big guys in the cloud hugging face of course absolutely legendary and you know there's just so much proliferation of these skills and tool sets that it's important to be able to sort of start building. And if you're going to build anything today, rag is the thing. You know, rag, this was put out by Langchain the other day in a deconstructing rag blog. Highly recommend this as a read to everybody out there. But the interesting thing too is that if you watched Andrej Karpathy's recent talk about the LLM operating system, RAG is sort of this piece. And I really love this sort of connection. So at a high level, if you're okay with prompt engineering, it's time to start getting okay with RAG. And that's what we're really focused on building out today. So what is RAG anyways? Let's talk about it. Let's think about these two core components it breaks down into. And let's think about how we can build RAG systems. Now, generally, you want to think of building LLM applications in three steps, and this is how you want to build today. You want to prompt engineer as far as you can. Then, if that's not working well enough, you want to introduce some additional context through your documents. and RAG systems as question answering systems in their most basic form. And these are going to be really helpful to getting our system set up to do the things we want it to do. If you're going to be a coach, if you're going to be a tutor, if you're going to be an assistant, you need to be able to answer questions very well, right? And so this is the big idea with RAG. When we talk about fine-tuning, there's a lot that goes into different parts of fine-tuning. That's a little bit beyond the scope of today is to cover all of fine-tuning. But it's important to understand we will cover fine-tuning embedding models. And it's sort of this new emerging paradigm where we're saying, okay, well, actually, it's not a linear thing anymore. It's not prompt engineering, then RA well, actually, it's not a linear thing anymore. It's not prompt engineering, then rag, then fine tuning. But rather, these are two different dimensions of optimization. One is optimizing the context, what the model needs to know. One is optimizing the actual language model, how the model needs to act. Now, you can say this is more of a two-dimensional thing. I think this is a useful mental model, but oftentimes you do end up still going, prompting, adding some examples, adding some RAG, then doing some fine-tuning. It does look linear in many cases, and it does get iterative between RAG systems and fine-tuning in the limit of trying to achieve human performance. That's for sure. Okay, so what is RAG anyways? Well, let's consider that RAG is question answering over documents. Let's look at what a final demo might look like, right? So this is a simple, the most simple RAG system you might build. This is called chat with your PDF. that is not where it's supposed to be. That is Isaac Asimov's iRobot. And if we take this PDF, we can potentially go ahead and ask this guy. We can ask this guy in, it's going to process it. It's going to set this up for us to be able to ask questions. In one or two word summaries what are the three laws of robotics it's going to use a conversational retrieval chain and by the way the live demo gods are already getting their fury out on me so hopefully they're kind to you by the end of the day today. And what we'll see is we do have the three laws of robotics. No robot may harm a human being. Robot must obey the orders. In one word each, what are the three laws? And we can sort of engage with this thing and get, based on our input and based on the retrieved documents and information, we can get some really interesting outputs, right? First law, protect. Second law, obey. Third law, preserve. Let's look at the source material here. We're seeing, oh, okay. So look at this. Self-preservation is the third law. Preserve, all right? Very fact-checkable outputs here. This is the big idea with RAG, fact-checkable outputs. All right, so this is as simple as we can get. And if we were demoing this, we can get. And if we were demoing this, we would probably add some additional next level pieces to it. What kind of next level pieces might we add, Wiz? Well, Greg, basically, we're missing quite a few things here. Number one would be better sourcing. So having the sources perhaps attached to the, you know, the actual PDF. Number two, we're doing very naive retrieval here. We're just kind of chunking and then retrieving. There's a lot of different kind of retrieval methods that we could explore to prove the application. We're not doing any real monitoring or evaluation here. We're just kind of plugging it in and away we go. We're using all closed source models. So we could extend this to an open source model. We can even use open source embeddings. We could then fine tune these pieces to provide us better generation and retrieval. Really, the sky's the limit here in terms of what we could do with this base application, but those are just a few examples. And those are the kinds of things that we're going to be talking about for the rest of the day today. So this is at a high level, the simplest possible RAG system. And what's the big idea here? Well, the big idea is that if you just ask LLMs questions, they're not necessarily fact checkable, and they will give you confident false responses sometime. So fact checking is very important. It's very important in life, right? Nobody likes fake news, and it's very important as we build AI applications, perhaps more important. So retrieval augmented generation is simply talking about going and finding references in your documents, adding those references to your prompt, augmenting the prompt, so to speak, and then improving your generations. Generative AI, right? That's it. Find them, add them, improve, rag. That's all there is to it. And when we talk about this being useful, one of the things it's very useful for is when you have very specialized domain language. So we had a few legal applications, people were thinking of building. This is true if you have legal jargon, if you're talking to real doctors, if you're working in the financial industry, government acronym soup, researchers have their own special language in each building at the university, and so on and so on. Also, put a pin in this. This is the place where you might want to consider fine-tuning your embedding model as well. So what is RAG? Well, RAG is, as we like to break it down, dense vector retrieval and in-context learning. The R in RAG comes first. So let's go part one on the R in RAG. RAG is, first off, dense vector retrieval. One of the most commonly misunderstood things is a classic NLP thing, the idea of turning words, text into numbers, embeddings. This is going to allow us to view words in an embedding space relative to one another. So this is an example we like to show from cohere's blog where would you put the word Apple can I get some answers in the chat here where does Apple go C crushing it Victor he's on it right by the bananas and the strawberry and the grapes cherries cherries Cherries? Cherries. All right. So what about where does cow go on this one? A. We got A from Manny. Anybody else? C from Taz. C from Victor. The answer is actually C here because puppy is to dog as calf is to cow. So you might actually say this is a dimension of age. Maybe this is a dimension of size or height or weight. And the interesting question is though, where does the legal rag system fit in on this chart? Where does the LLM wizard fit in on this chart? Where does the Millennium Falcon that Sarah is sitting in right now fit in on this chart? Millennium Falcon that Sarah is sitting in right now fit in on this chart it's not clear we need more dimensions of embedding space this is the idea of many many dimensions of embeddings let's say over 4 000 in some cases again from cohere here just to represent a single word relative to lots of other single words of course you can do embeddings at the token or subword level, and you can also do embeddings at the sentence level, as we will show later today. But the most classic example, of course, is from 10 years ago now, 2013, Word2Vec comes out. Word2Vec, you get it, right? And we can do classic things like king minus man plus woman is about queen. That's when we know our vectors are kind of working. And so the idea is that this, even back then, they called out, they said, well, this is really useful for many downstream tasks. We obsess over this language today with LLMs. It's very important to focus on downstream tasks. We obsess over this language today with LLMs. It's very important to focus on downstream tasks. And Word2Vec is one of the most iconic and earliest examples of dense vector representations. Dense vector. Today, we have much better embedding models, like OpenAI's Ada, which we'll be getting started with today. You can also find the latest and greatest, that new hotness on the massive text embedding benchmark leaderboard from Hugging Face. So when we think about now putting the R in RAG part two, now what we're talking about is we're talking about retrieval. We got the dense vector idea. Now let's talk about retrieval. Here, we're going and we're putting a query in to an embedding model. That's going to convert that query into a vector. And then we're going to go to a vector database and we're going to look for stuff like it. Right? So the big idea here is there's three easy pieces to retrieval. You ask a question, you search the database for stuff that's similar, and then you return the stuff that's similar. And when we talk about now creating that database or creating that index, shout out Lama Index, right? We're often talking about as we get started, there's only one type of index we need to care about. It's called a vector store or a vector database or a vector store index, or just an index. There's lots of ways to talk about it. We're taking the docs, we're chunking them up, we're pushing them through an embeddings model, and then we're creating our vector store. Why is that important? Because then when we go to actually ask a question, we push it through our embedding model, we get a vector, and we look through a cosine similarity generally to find nearest neighbor similar things within the database, within the vector store. So why is it dense vector retrieval? Because we ask a question, it's a vector. We search the vector db for stuff that's cosine similar, get it back. Then we return the stuff we found in natural language, right? So we've got our query. We go to find similar stuff. We're setting up now our prompt template. That's what we're going to talk about next. It might say something like, use the provided context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer or cannot answer, please respond with, I don't know. that it was directly in the prompt, the prompt playground that we were able to see this exact prompt that we used in the Chainlit application. So then we go ahead and we take our similar context from the vector database and we shove it right in to the prompt, reference one, reference two, reference three. We saw this before. Finally, all that stuff is in the prompt. We shove it into the LLM. We get our answer. Bada-bing, bada-boom. So this prompt template piece, this piece is containing this idea of in context learning where does in context learning come from in context learning comes from the classic paper right language models are few shot learners this is GPT-3 during the release they introduced key concepts like one-shot learning, few-shot learning, in-context learning, here we are, and meta-learning, which is pretty meta. We're not going to get into it right now, but definitely check it out. It's definitely worth understanding if you're into these sort of prompt tuning things and other things that are sort of half out, half in the model. So scaling up models greatly improves task agnostic few shot performance what are we talking about well this is an image from the paper and the image shows that with more examples in context we get better accuracy this is representative of many types of tasks better accuracy. This is representative of many types of tasks. And the big idea is that larger models make increasingly efficient use of in-context information. We're simply talking about prompt engineering here. With zero shot, that means instruction only, right? translate the following word. We're not doing so hot, but with just one example on a large enough model, we get massive improvement. Between, let's say, 10 and 100 examples, or 100 and 1,000 examples, we're getting much, much better performance. we're getting much, much better performance. So the idea is, you know, give examples in your prompt, give a few demos, right? Use the input to the LLM to tell it about the kind of task you want to do. So you should always start here and you should expect it to complete further instances. You can make up words like what, poo, and fart, uddle here. And it's very fun to do this sometime. You might consider doing something like this in your demo today. What we're talking about here is we're talking about a spectrum, though. We're talking about how much task-specific data do we need. I was talking to an enterprise yesterday, and he said, well, I was sho to an Enterprise yesterday and he said well I was shoving as many examples as I could into the prompt and then they finally increased the context window size and I could shove more in but I still can't shove them all in it's like you're on the path guy you're on the path because this is the path zero shot to one shot to few shot where does it lead well inevitably it leads to once you get to thousands of examples, it leads to traditional fine tuning. This is a spectrum that ends in some measure, kind of where we began the day today, right? And so what we're talking about when we're talking about in-context learning is this is just, of course, prompt engineering. When we talk about prompt engineering, the most important thing is to be clear and specific in your instructions. It's to give it a specific context, a place from which to stand, very clear rules that you want it to follow. If you don't know the answer, don't say, I don't know. And context also includes in RAG systems, things like reference material that we found. So as we give input, we want to provide instruction, zero shot, an example, one shot, maybe 10, 100 examples, few shot. And we want to give really well thought through examples often too. If we want to build agentic systems that are much more interesting and connected to many other things, it's often very, very important to specify the output. Shout out to OpenAI's recently released JSON mode. This is something that's been up and coming in the space for quite some time. So when we talk about taking this to the next level, you want to make sure that you want to give really good examples that have that chain of reasoning within them. This is the idea of chain of thought prompting. We're not going to dig too deep on this, nor are we going to dig too deep on the idea of moving from standard to chain of thought prompting through this idea of self-refinement. But I would encourage everybody to check out these papers. This is the idea of, yo, model, can you do this? It's like, yeah, sure, Greg, here it is. And it's like, I don't know. Can you tell me exactly how you got that answer? And this is the simple way we can interact to improve our ability to prompt in the future. So what have we got? We've got RAG is dense vector retrieval plus in-context learning. Looks something like this. And in conclusion, we can use RAG to find references, stuff them into the prompt, augment the prompt, generate better outputs. Components include vector store, aka an index, aka a vector database. They include embedding models and LLMs and they include doing some prompt templating, aka prompt engineering. And with that, we're ready to move on to the next piece of our chativersary content, how to build with Langchain.
Session 2: Retrieval Augmented Generation (RAG), An Overview
What you'll learn in this session:
- What is RAG, exactly, and why do we need it?
- RAG = Question Answering over Documents
- Retrieval Augmented Generation = Dense Vector Retrieval + In-Context Learning

Speakers: 
Dr. Greg Loughnane, Founder & CEO AI Makerspace.
https://www.linkedin.com/in/greglough...

Chris Alexiuk, CTO AI Makerspace.
https://www.linkedin.com/in/csalexiuk/

Apply for one of our AI Engineering Courses today! 
https://www.aimakerspace.io/cohorts