 Hey this is Lance from Langchain. So Meta just released LLAMA 3.2 today, which is really exciting. I've been looking forward to this for a while. And what's really neat is you can use increasingly small models to do pretty interesting things locally on your devices. So today I'm gonna show how we can go from scratch, build a fairly complex RAG agent using a 3 billion parameter model that will run locally on my laptop. And this diagram shows the entire flow of the agent. I'll just zoom in here. So we're going to be able to take questions, route them using the model to either an index or web search, grade documents from retrieval based on relevance to our question, produce an answer, grade the answer for hallucinations, grade it for relevance to the question, and finally return an answer, grade the answer for hallucinations, grade it for relevance to the question, and finally return an answer to the user. We can do all that logic locally with LAMA 3.2 3 billion parameter model and I'll show you how to do that shortly. So first there's kind of two variants of the models here. There's compact and there's vision models. So there's a 1 billion and 3 billion parameter compact model. We're going to be working with a 3 billion. There's also vision models which are interesting for other reasons and which I may talk about in a future video. So again these compact models are meant for smaller on-device use cases. And what's kind of cool is I show some statistics here comparing the prior 8 billion to the current 3.23 billion and it's pretty close. Three versus eight, you can see the three billion parameter models as strong on one of the evals. And so it's pretty promising. It's telling us that smaller models are getting indeed better and that opens up access to more folks which is very exciting. So I want to talk a little bit about this RAG agent we're going to build and let's just get into the code. So we're going to incorporate ideas from three papers. One is routing. So that's the ability to take a question and route it to either a vector store or web search based on the content of the question. The other is fallback, this idea that you can basically retrieve documents and grade them so if any documents are not relevant you can filter them out. A big problem with retrieval is erroneous retrievals and so we have a fallback for that. And self-correction, we can fix and catch hallucinations and we can basically reject answers that don't address our question. We can build all that from scratch and I'll show you right now. So I'm going to be moving between a notebook. I've already set this up and this code will be available to you. I've just done a few pip installs and here's where some interesting decisions can be made. I'm going to use local embeddings from gnomic, these GPT for all embeddings. The pip installs are presented up here. You can try other local embeddings. I tend to like these, but obviously that's up to you. And I'm going to access LLAMA 3.2 Instruct FP16 from OLAMA. You can try other providers to access local models. I happen to like OLAMA. So if you go to OLAMA's website here, they provide a bunch of different models that you can just download. And all you need to do is just Olama pull and then whatever model you want. So that's all you need to do. Very simple, just make sure you have Olama installed and you can access any of these models for free. They're pretty cool. So I've done that and now there's two other pieces of tooling and then we can get right into it. I use Tavoli for web search. I'm going to set up this kind of web search component of my agent. And Tavoli is a really nice search engine optimized for RAG and other agent use cases. It has a generous free tier so you can sign up and use it. Simply, they'll give you an API key. Just make sure it's set. I'll also use Langsmith for tracing, which is just a way I'm going to visualize what I'm doing. This is totally optional. So those are the only things I've set here. And then we have a blank notebook. We can just get going. First, I'm just going to find my LLM. So again, I set the name of the model that I pulled using this command, olama pull. This is now available to me. And I'm using LangChains integration with olama. So I I just simply specify this model name and I load it. Now you see I do two interesting things here. First I load just the raw model and I also load this JSON mode and you'll see why we're going to use that a little bit later. Next I'm going to build a vector store. So this is where it's kind of up to you. This could be your personal documents, this could be really anything you want to work with for any question-answer application locally. In my case, I'm just going to take three URLs from some blog posts that I like, and you're going to see here this is just some very simple utilities to load the documents using a web loader, split them as I do here. You can select your chunk size arbitrarily. So what's happening here is we're loading the raw documents. We're splitting them into chunks of size a thousand tokens, and I'm going to add them to a vector store. This is a free to use vector store from sklearn. I load in my document splits. I load in my embedding. Now here's where I'm going to be using those gnomic local embeddings that again are free to use and I'm going to create a retriever from that. Let's get that started. So this retriever is a line chain abstraction that gives you a few common methods. So you can use invoke to very simply agent memory get documents related to any input. Here we go. So we've just retrieved some documents relevant to this question or statement agent memory. Of course, what's happening is it's taking the statement, embedding it using our model, and then doing a similarity search with the embedded documents to find similar matches based upon semantics of my question relative to the documents. That's it. So we get documents back. So now we have a retriever. The retriever you can see falls into our flow here. This is kind of one of the core components of a RAG system. You need some document retrieval mechanism and we built that. So now let's build some of this logic around it. First let's start with routing and I'll show you how we can do that. So now I'm going to build a router and I'll show you a really simple trick for doing this with local LLMs like this. Very simply, I'm just going to set up a prompt that says you're an expert at routing. The VectorStore contains documents related to agents, prompt engineering, and adversarial attacks. Use the VectorStore for questions on these topics. For everything else, especially current events, just use WebSearch. Now, here's where I just instruct return JSON object with a single key data source that is either web search or vector store, depending on the question. Now, when I use this LLM JSON mode, if you go up here, right here, because I've set format JSON, it's going to enforce the output of the LLM to be JSON. So what's nice is I've built myself a simple little router just using kind of LLM and JSON mode. So all I need to do is I'm going to pass in a question. Now, one other thing I want to mention, this is a chat model. So we can pass a set of chat messages. You can import from Langchain core messages, human message, system message. This is my system message with the router instructions. And then my human message is just whatever question I have. What are the types of agent memory? And those can be passed in as a list. That's it. And I can go ahead and run that, oh, the data source for this is VectorStore. And I get a little JSON back that says DataSource VectorStore. So you build a simple router using JSON mode and a local LLM. And I use this all the time. It's an extremely convenient way to just do really simple kind of binary routing using the llm to reason about some input and return a structured object like a json with for example the two outputs that i want either vector store or web search so now we've built this red router piece and next i'm going to build this blue greater piece so if i go down here it's also really simple let's just give some instructions for grading. The main idea here is I want to look at the retrieved documents and determine that they're relevant to the question I ask. Because sometimes with semantic similarity search, you can get kind of erroneous retrievals of documents that are not particularly relevant to the question, but happen to be returned based upon the idiosyncrasies of your chunking or your embedding model. So this can happen sometimes. It's nice to have this additional layer of kind of protection or reasoning on your retrievals. So you're greater assessing the relevance of retrieved documents. If the document contains keywords or semantic meaning related to the question, grade it as relevant. Okay, so here I'm just going to pass in the document, the question, think carefully about this, and return a JSON with a binary score, yes or no. That's it. So here we go. I'm just going to ask a question. What is chain of thought prompting? I'm going to retrieve documents from my vector store. I'll pick one of them. Okay, you can see that's what I'm doing right here, and I'll go ahead and pass that into my prompt right here. I just format the string with those two inputs. And I do just what I did before. I'm going to go ahead and pass in those greater instructions here. I'm going to pass in my formatted prompt here, which has my question and my document. There we go. So we get binary score yes, which makes sense. Basically, what is chain of thought prompting? We can look at the document to confirm that. And there it is. So the document contains chain of thought prompting here. So it mentions it in a few places, so it's relevant to our question. That's good. So now we've built the router, we've built this particular document grader. You can see we've done this really simply using a chat model and JSON mode. That's all we need to build these logical building blocks. Now let's just show you how to do really simple RAG. Here's just a RAG prompt. You're an assistant for question answering. Here's some context. Think about it carefully, review the user's question, and then provide an answer. That's all I do. I retrieve some docs. I just format them really simply, just to pull out the page content from each document object, and I format my prompt accordingly, just here, just string formatting, and then I just go ahead and pass that as a message into my chat model. In this case, I don't need to use JSON mode because I want natural language to string output. I don't need a JSON object. Let's try that out. There it is. So this is like a nice answer about chain of thought prompting. So we've actually already done quite a bit. We have a router. We've shown how to do document grading. Now we've shown how to generate an answer. Pretty nice. This web search thing we actually already have. We use Tavoli for that. We have the web search tool defined down here, so that's cool. I can show you how we would use that really quickly if I go down here. So this, all I need to do is just invoke it, invoke, and I can pass in a question. And I get some documents out that are related to LLM agents and it gives me URLs and all that. So this is pretty nice. We'll be using this later. So now let's just show these final two components, hallucination grading and answer grading. So it's going to be really similar we already did. We'll copy this over. So just some hallucination grader instructions, just kind of logically the same as this but with slightly different prompt but same same concept, right? You're a grader teaching a quiz, grading a quiz, you'll be given some retrieved documents, you'll be given an answer, and you're basically saying, hey, grade to see if the answer is ground in the documents, that's all we're gonna do here. So here is the final prompt, this is where I'll put the documents, this is where I'll put the generation, this is where I'll put the generation. And I'm just going to format it using the document that I already retrieved, which is here. That's all my doc text. And the generate answer from above. That's this. And let's just run this greater on it and see. Cool. So whatever the nice thing, I add basically this whole explanation in the prompt. So I basically say, hey, give me a grade and explanation. In this case, you can see why it graded. So if we step back, what's kind of cool is this is all being done on my laptop locally using a 3 billion parameter model, and it runs pretty quickly. So, you know, we've already implemented quite a bit of logic here. We're through the grading hallucination bit and there's one more grader. That's just the final answer grader. This is going to be similar, but it's going to look at the final answer and it's going to look at the question. It just checks, hey, does the answer actually address the question? So that's really it. Let's go ahead and run that. Here I have a test case. I basically pass in a question. I pass in an example answer that I got from one of the earlier tests of the system and I just run that. Cool. So I get a score and I get an explanation. That's great. So we've actually already done really most of the hard work here. We've built all these components individually. We've shown how to use JSON mode to build a router. JSON mode, how to build graders for documents, hallucinations, and answer relevance. We've just used an LLM to produce answers using RAG. We have a web search tool, and we have a document grader tool. And we have a retriever. So we have a document grader tool. And we have a retriever. So we have all these interesting components. Now here's the question. How do you actually orchestrate this flow and tie them all together? And that's where I want to introduce this idea of LandGraph. So LandGraph is an agent orchestration tool that we built. I use it quite extensively. There's other videos on it. And I just want to show again how you can use it to orchestrate this whole workflow really easily and running locally on my laptop. So the first part of line graph is actually defining some state that's going to live across this entire agent flow. So the state is basically going to be a set of things, this can be like a schema, that each of these nodes has access to. Now because we're doing RAG, it's going to be things that are intuitive to the RAG process. It's going to be like the input question, the retrieve documents, the answer. All the things that I input and output from each of these steps individually, I'm going to save to this common state. And that's going to persist throughout my agent's lifetime in LandGraph. So what's cool is in LandGraph, I can go ahead and define this state class. And this is very flexible. I use a type dict for this. Here's the keys. These are things can be relevant to my agent's behavior and here is a type pin for each one. So that's all I need. That's kind of the first component of my agent. Now the second component of my agent is basically taking each of these steps that we did individually and wrapping those all as individual functions. Okay, those are known as nodes. So those functions really all do one thing. They take state as an input and they modify it in some way. So let's look at an example. If I go down, I look at our router, for example. What did our router actually do? It took in the question and it output where to go next. Go to a vector store or go to web search, for example. So this function is, for example, an edge in our graph. So it will basically look at the question and route it to one of two places. So edges provide some connectivity between the nodes in our graph. Nodes, on the other hand, if you look at retrieval, for example this, this takes in a question and returns documents. So for retrieval, this is what I call a node because in this case we're actually modifying the state, we're adding something new to it like documents, and in edges we're just deciding where to go next. Those are the two concepts they're going to be using here. Nodes and edges. Edges kind of direct where to go. Nodes modify the state. Now I'm going to show you that all right now. So here's some code that lays out the nodes and edges of the graph using the things we already defined above. So we're going to find a node for retrieve. That's going to take our retriever, invoke it with a question, the question we get from state, the state we pass into the node, and the state is just this thing. The state's just this dictionary. This dictionary has a key question, which we can just reference easily, as we see here. Get the question from state, invoke the retriever with the question, get documents. Now see this? We just can write the documents back to state using a dict, just like we do here. So in this case, documents then overwrite the value of documents here in the state. That's really it. Same idea for generate. You saw above that we actually ran. Same with generate. So if you look at this logic here, this is exactly what we did above. Here. So this is actually where we ran and tested this in isolation. Now I'm just wrapping this functionality in a function called generate, which is going to now grab from our state question documents and loop step. This is going to track the number of generations that I do. And it's going to write back the generation and increment the loop step, write those back out to the state. That's all that's going to happen. Now we're at grade documents. So if you kind of look at our overall flow, grade documents is this piece. We've done retrieval. That was the first node we defined. Grade documents is this other node. And in great documents we're going to do, this is exactly what we did above, but in this case we're just going to grab the documents that we had in state, which were written to state by a retriever. So if I back up a little bit, we've retriever node that takes in a question from state, retrieves documents, writes those documents back out to state. Then the grade documents node takes in state and has access to the documents that we wrote and does something else on them. In this node, it will basically look at each document, grade it for relevance to the question, this just like we did above, and here's where we can add some logic. So basically we will keep it if it's relevant. If it's not relevant, we're going to set this flag to do web search, and we're going to at the end of this process write the new value of our web search flag and the filter documents back out to state. So that's really all we're doing in these nodes. We're basically taking the state, looking at it, reasoning about it, writing something back out to state. That's all that's happening. Web search is another node. We basically start here. We take the question. We take any existing documents that we have. We do web search using the retrieval tool, and we write the documents back out to state. That's all that's going on. Now we're almost done with this process. Here's where we define our edges. So the edges define the logical routing between the nodes. So if you go back to the start of our graph, we had this router. The router is just an edge. It's conditional edge that looks at the question, decides either to go through the retrieval node or go to the web search node. And so you can see we invoke our router just like we did above. We get the source like where to go. If it's web search, return web search. If it's vector store, return vector store. You'll see a little bit later how we use that. But in these edges, you can see what's going on is we're returning a node to go to where web search is our web search node, vector store is our vector store node. So the way to think about this is in nodes, I take in state and I update state. In edges, I take in state and I decide what node to visit next. That's all that's happening. Decide to generate. This one is going to basically decide if there is web search flag that's been enabled. I go to web search. Otherwise, I go directly to generate. And that matches exactly what we show here. That's kind of what this edge is. So basically, this is indicating I either go to web search if any documents have been deemed irrelevant, or I go to generate. Now, this is the final step in our flow where we do grading for hallucinations and answer relevance or answer usefulness to the question. So this is a hallucination check. We already showed that above. We're just adding it to this edge and we're going to get a grade. So check hallucinations. If the grade is yes, as in it's, it is not, there's no hallucination, it's grounded in the documents. Then we do the grade generation versus the question. So that's the second check. We already showed that above. That's what we do right here. And then if that's yes, then we deem the result is useful. So that is that the answer is actually useful to kind of finalize and send to the user. Otherwise, the answer is deemed not useful. Okay. Now you can also see we do this max retry check here. So if you recall in the generation node, we incremented that every time we do generation. We want to make sure we don't keep retrying. So we basically check this max retry. So if you default to three, and if that's exceeded, then we just will exit. And that's what you can kind of see here. MaxRetrize is reached. So this looks at basically if there was hallucination present, then the answer is not supported by the documents. And then finally, MaxRetrize again. So that is the final edge. Now we're going to add logic to stitch this all together. So this is where I'm going to add all the nodes I defined above with names and I'm going to then take all the conditional edges which I defined basically here, route question for example, this returns either web search or vector store right so basically this is where I can map the outputs of my edge to the node names in this case I make it simple because I keep them the same but you can imagine those can be different. So basically this is the mapping between the node to visit and the output of the conditional edge, but that's really it. So I add a conditional entry point, I then add an edge from web search to generate, and so I create the connectivity of my graph here, I compile it, and I can visualize it, and there we go. That's our entire graph, visualize, it's pretty cool. So think about all we're doing. We're starting with that conditional edge that's going to look at the question, decide to either visit web search or vector store based on the question. That's that router we built first. Then we're going to do retrieval. We'll grade documents. If any documents are not relevant, we're going to go ahead and do web search and supplement them with web search. Otherwise, we'll go to generate. So from web search, we'll go to generate. Now when we've done with generate, we did two checks. One is that the generation is not supported by the documents, so there's hallucinations. In that case, we just retry. That's what this is representing. The other is that it's actually not useful, so it doesn't actually answer the question. If that occurs, we kick back and retry web search. So that's kind of the current logic in the graph, and you can modify this accordingly, but that's what's going to go on. So those are what the two fallbacks do. And then finally, the answer is either useful, which means that it's both supported as a no hallucinations, and it's useful as an answer to the question, or max retries. That's another way we end. But that's the overall connectivity of our graph we've put together in line graph. So we zoom all the way back out. We've used Ollama. We've used Ollama to load Llama3b FP16, LLM and JSON mode. I've tested each of these components out individually. I built a router, built a retrieval grader, built a rag prompt, built a hallucination grader, built an answer grader. We tested all of them individually. Now I put them together into a graph using line graph. Here's the overall graph connectivity and we can test this now. So when I kick this off, all I need to do is supply some initial state values to work with. So basically, I can pass in a question, and I'll pass in maxRetrizes3. These will just populate my state with those two inputs, and that's all the graph needs to get started. Now, I just can run graph.stream. That's one way to run the graph. Pass in the inputs and I'm going to stream in the graph state at every step and I'll print it out so we can see this running right now. So first we're going route question. It's going to decide whether to use the vector store or web search based upon the content of the question. So it decides the vector store, that's good. It retrieved these documents, so it's basically spitting out everything here, so it's writing up the whole state every time. So this is cool, it's already done a bunch of work here, it's gone ahead and it has graded the documents, it deemed one of them to be non-relevant but the rest are relevant. So then it decides we'll go to web search, it's going to be non-relevant, but the rest are relevant. So then it decides, we'll go to web search. It's going to be kicking off web search. It runs that. This is the entire state printed out by web search. Cool. And basically it did the hallucination check and it also graded the generation relative to the question and that's it. So the entire graph ran. Now I can go over to Langsmith and show you also the entire flow of that graph. So here I'm in Langsmith. I'm looking at the trace. So you can see, first, we do retrieval from the VectorStoreRetriever. We get a bunch of documents. That's cool. We graded them for relevance. I can open that up and we can see here is Chet Ollama grading each document. Pretty nice. And here is what I pass and here's the system prompt I pass. Here is the human message. Pretty cool. Which contains all the documents and here's the rendered output. It deems that to be relevant. It goes through each one of the four. So that's cool. Now here's where it basically hits that conditional led to decide to generate. Because one of the documents is deemed to be not relevant, it decides I want to do a web search. So then it kicks over and goes to the web search node. That runs Tavali. That returns some web search results for me. Pretty nice. Then I go to generate, which runs Chattelama again. In this case, it generates an answer to my question. My question was related to agent memory. There are two categories of agent memory, short-term and long-term, and so forth based upon web search and my retrieval. It answers the question. So then this is the conditional edge that will both grade the hallucinations. So in this particular case, it deems that the answer is grounded. So that's cool. I can look at the grading here and it finally graded you can see here yes for answer helpfulness. So it deems that it's actually helpful and actually addresses the question. So that's the entire flow of our agent that does a whole bunch of different things, all running locally on my laptop using a 3 billion parameter model. So it's pretty neat. If I go back and kind of show the notebook, I'll make sure this is all accessible so you can play with this yourself. This is all free to use. I'm using local free embeddings. I'm using locally running 3 billion parameter model. I'm orchestrating this using Langraph. You can orchestrate things that are much more complicated or much less complicated, but this is really showing you that even with a 3 billion parameter model, you can build pretty complicated, interesting multi-step workflows that actually run reliably and will run locally. So again, I've actually been really excited about working with local models for a while. I find that orchestrating them with LandGraph is a very nice way to kind of build these reliable multi-step workflows that can run locally on my laptop to do arbitrary tasks. This is just a toy example I'd like to use to showcase the functionality in a RAG setting. But it can handle many other types of tasks you can imagine running locally and also reliably as we just saw here. This is an agent that would be fairly difficult to run using a more react style open-ended agent loop but using lang graph with each of these nodes and edges very clearly enumerated you're asking the LLM to make very specific narrow decisions at specific points as you can see here using json mode like we did in the router. And if you structure it that way, you actually can build pretty complicated workflows that actually run liably and locally. So please leave any questions in the comments. Happy to address them, and I'll make sure this is all open source. Thanks.
Reliable, fully local RAG agents with LLaMA3.2-3b
LLaMA3.2 has released a new set of compact models designed for on-device use cases, such as locally running assistants. Here, we show how LangGraph can enable these types of local assistant by building a multi-step RAG agent - this combines ideas from 3 advanced RAG papers (Adaptive RAG, Corrective RAG, and Self-RAG) into a single control flow using LangGraph. But we show LangGraph makes it possible to run a complex agent locally. 

Code:
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag_local/

Llama3.2:
https://huggingface.co/blog/llama32#what-is-special-about-llama-32-1b-and-3b

Full course on LangGraph: 
https://academy.langchain.com/courses/intro-to-langgraph