 Hi, this is Lance from Langchain. We're starting a new series called RAG from Scratch that's going to walk through some of the basic principles for RAG and kind of build up to advanced topics. that LLMs haven't seen all of the data that you may care about. So like private data or very recent data would not be included in the pre-training runs for these LLMs. And you can see here on the graph on the x-axis that the number of tokens that they're pre-trained on, which is of course very large, but of course it's still always going to be limited relative to private data that you care about or, example recent data but there's another interesting consideration is that lms have context windows that are actually getting increasingly large so you know coming going from like thousands of tokens to many thousands of tokens which represents you know dozens of pages up to hundreds of pages we can fit information into them from external sources. And a way to think about this is LLMs are kind of a kernel of a new kind of operating system. And connecting them to external data is kind of a very central capability in the development of this new emergent operating system. So retrieval augmented generation, or RAG, is a very popular kind of general paradigm for doing this which typically involves three stages. So the first stage is indexing some external documents so they can be easily retrieved based on an input query. So for example we ask a question, we retrieve documents that are relevant to that question, we feed those documents into an LLM in the final generation stage to produce an answer that's grounded in those retrieved documents. Now, we're starting from scratch, but we're going to kind of build up to this broader view of RAG. You can see here, there's a lot of interesting methods and tricks that kind of fan out from those three basic components of indexing, retrieval, and generation. And future videos are actually going to walk through those in detail. We're going to try to keep each video pretty short, like five minutes. But we're going to spend a lot of time on some of those more advanced topics. First, over the next three videos, I'll just be laying out the very basic kind of ideas behind indexing, retrieval, and generation, and then we'll kind of build beyond that into those more advanced themes. And now I want to show just a quick code walkthrough because we want to make these videos also a little bit interactive. So right here, and this repo will be shared, it's public. I have a notebook open, and I've just basically installed a few packages. And I've set a few environment variables for my Langsmith keys, which I personally do recommend. It's really useful for tracing observability, particularly when you're building RAG pipelines. So what I'm gonna show here is the code for our RAG quick start, which is linked here. And I'm gonna run this, but I'm then gonna kind of walk through everything that's going on. So actually, if we think back to our diagram, all we're doing here is we're loading documents. In this case, I'm loading a blog post. We're then splitting them. And we'll talk about that in future short videos on why splitting is important. But just for now, recognize we're splitting them or setting a chunk size of 1,000 characters. So we're splitting up our documents. Every split is embedded and indexed into this vector store. So we said we picked open AI embeddings, we're using Chromas, our vector storage runs locally. And now we define this retriever. We then have defined a prompt for RAG. We defined our LLM. We've done some minor document processing. We set up this chain, which will basically take our input question, run our retriever to fetch relevant documents, put the retrieved documents and our question into our prompt, pass it to the LLM, format the output as a string, and we can see here's our output. Now we can open up LangSmith and we can actually see how this ran. So here's our question and here's our output, and we can actually look here's our retriever, here's our retrieved documents, so that's pretty nice. And ultimately here was the prompt that we actually passed into the LLM. You're an assistant for QA tasks. Use the following pieces of retrieved content to answer the question. Here's our question. And then here's all the content. This we retrieved. And that drills in our answer. So this just gives a very general overview of how RAG works. And in future short videos, we're going to like break down each of these pieces in a lot more detail. Thanks.
RAG From Scratch: Part 1 (Overview)
LLMs are a powerful new platform, but they are not always trained on data that is relevant for our tasks. This is where retrieval augmented generation (or RAG) comes in: RAG is a general methodology for connecting LLMs with external data sources such as private or recent data. It allows LLMs to use external data in generation of their output. This video series will build up an understanding of RAG from scratch, starting with the basics of indexing, retrieval, and generation. It will build up to more advanced techniques to address edge cases or challenges in RAG. 

Code: 
https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_1_to_4.ipynb

Slides:
https://docs.google.com/presentation/d/1C9IaAwHoWcc4RSTqo-pCoN3h0nCgqV2JEYZUJunv_9Q/edit?usp=sharing