 Hey this is Lance from Langchain. We seem very high interest in building LLM agents using open source LLMs and so we wanted to talk through how to do that from scratch using LLAMA3. So first what is an agent? So Lily and Wang has a very nice blog post that laid out the central components of agents being planning, memory, and tool use. So I want to walk through these components individually and how I can use them with LLAMA3. So first let's talk about tool use. I'm going to copy over some code and we're going to walk through it. So I want to walk through these components individually and how I can use them with Lama3. So first let's talk about tool use. I'm going to copy over some code and we're going to walk through it. So I have this notebook, done a few pip installs, set a few API keys. We'll use Grok as our LM. We'll use Tavoli for web search as one of our tools. And we'll use Lanksmith for tracing. But that's all I've done here. Okay. And I'm going to kind of have this image side by side so we can look at it. So first tool use. What's the big idea here? The big idea is simply this. I want to take an LLM, give it awareness of some external tool that exists, and have the LLM return the payload necessary to invoke that tool. That's really all that's going on. Now this is often kind of confused and I wanted to kind of zoom in and explain this exactly. So let's say I have a function called magic function which takes an input and adds two to it. I want to give an LLM the ability to recognize whether or not to invoke this function and to return the payload necessary to run the function given the user input. So here's exactly what I want to have happen. I want to take that function, somehow bind it to my LLM, and given an input, then return both the function name itself and the arguments necessary to run the function. Remember, LLMs are just string to string, right? It doesn't have the magic ability to call that function natively. But what it can do is return, okay, I've seen this function, I know it exists, and I'm going to give you exactly the input format necessary, the payload to run the function, as well as the name of the function. So that's really all that's going on. So first, this tool decorator in Langchain allows you to take any arbitrary function and just turn it into a tool. And let's just kick this off. So here's my magic function, and here's a web search function. So these are two things that I want to kind of turn into tools. And I can do that right here. So we can run this. Now if I look at magic function, now it's a structured tool. It has a name. It has a description. And it also has that input or arguments as captured as a pedantic schema. Okay. So all this information can be passed directly to our LLM. That's the key point. So this allows us to go from arbitrary functions to tools that can be bound to an LLM. Okay. So that's kind of step one. Now, step two, this is where things are kind of interesting. I'm going to use Grok here, and I'm going to use a prompt. I'm basically going to say, you're a helpful assistant with two tools web search and a custom function. Use web search for current events use the magic function if the user directly asks for it, otherwise just answer directly. So that's kind of my prompt. And let's test this in two cases to explain exactly how this works. So all I'm doing, I'm using chat grok, setting Lama 3, 70b and I'm creating this runnable. This is kind of a langchain primitive for basically invoking LLM. So that's all I've done. Now, here's what's interesting. This is piping the prompt to an LLM, and I've bound my tools to the LLM. So this is automatically taking those tools we defined, and it's basically giving them to the LLM so that it's aware of them. So that's basically represented in this red box here. You take external tools and you basically bind them to the LM so the LM is aware that they exist. That's kind of step one. Now here's step two. I can basically take a question. So I'm going to ask what is magic function three. I'm going to invoke my runnable or my chain, right, with this. And let's see what happens. I'm going to run this.nable or my chain with this. And let's see what happens. I'm going to run this. Now, here's what's interesting. That payload contains an object tool calls, which contains the name of the function and the arguments. That's it. So that's the key thing. And I can look at the raw payload as well. So the raw payload is just simply this AI message. It contains a bunch of information. But here's the main thing. It contains basically the name of the function to call and the arguments to pass to the function. So again, that's exactly represented here. All that's happening is I've taken a function, I've turned it into a tool, I've bound it to my LLM, I can ask a question in natural language, and the LLM can respond directly with the function to call or the tool to use and the input argument to use based upon the user input. That's the key point. And that's really all that's happening with function calling. That's all I need to know. Okay. So here's the other key thing. What if I just ask a question about the United States based on my prompt, it should not try to invoke any of these tools. Now let's test that. I run this. Good. And so this payload tool calls is empty. I can look at the raw payload. And yeah, now it's just a chat response, right? The capital of the US is Washington, DC. Great. Okay. So that's it. So hopefully now you understand how tool use works. And now remember, this requires an LLM that's actually been fine-tuned or prompted or otherwise is compatible with tool use. And this is a very important point. We talked to the folks at Kroc. They have kind of a proprietary implementation for how they do this, which we don't know fully. But it is reported that works very well with LLAMA-70B, LLAMA-370B. And in my experience, I've seen it to indeed work quite well. So in any case, the key point is this. I can take any arbitrary functions I want, I can turn them into tools, I can then pass those tools to an LLM, I can bind them, and then you can see right here, when I invoke my LLM with a question, the LLM makes the decision to use one of the tools, and if it does, it's going to return to you the name of the tool it wants to use and the input argument. That's the key point. Okay. So that is really what you need to know about tool use. Now we get to the fun stuff. We're gonna build the agent. And for this, I'm gonna use LandGraph. And I'm gonna explain kind of how this works over time. But first, the way to think about LandGraph is basically it's a way to lay out flows and flows in particular land graph are often characterized by cycles the ability to kind of do feedback and that's really relevant for agents and we'll explain why here shortly so land graph basically takes a state which can live over the course of your graph or flow and it can be accessed by all kind of what we're gonna call nodes in your graph okay so first, so first as state, I'm just going to find a set of messages. And don't worry too much about this for now. This will all make sense in about a minute. Okay, now here's where things are going to get interesting. I'm going to define an agent that contains two nodes. Okay, so first we're going to take our input. Again, it's a human message. We pass that to our LLM, which has the bound tools. The LLM is going to make a decision to use a tool or not, and we just walk through this. That's a step one. That's this thing we've already seen, right? Now what we're going to do in LandGraph is we're going to add basically what we're going to call a conditional edge. So this edge is going to, all it's going to do is say, was there a tool call or not? If there was a tool call, I'm going to route that over to a separate node that basically runs the tool. So let's walk through with our example we just did. What is magic function of three? The LLM made the decision to invoke the magic function, and it gave us the payload, right the payload right we just saw that so that's arguments input is three name is magic function those get plumbed over to what we're going to call tool node which actually invokes the necessary tool so it's going to basically take in this name magic function it's going to look up magic function itself and it's basically just going to run that function with this input payload, and then it's going to return that as a tool message to the LLM. That's all it's going to go on. LLM is going to see that tool message. It's going to make a decision about what to do next, and eventually this is going to keep running until there's a natural language response. And in this kind of toy example, the tool message would return with the result of five. That would be returned to the LLM. The LLM would see that and say, example, the tool message would return with the result of 5. That would be returned to the LM. The LM would see that and say, okay, the result is 5, and then you would exit. So that's like the toy example we want to see. Now we can implement this all in line graph really easily. And let's actually just talk through that quickly. I've copied over the code here. So all we've defined here is we have this assistant. So this is basically just wrapping the chain that we defined up here, this assistant runnable. We just wrap that. And basically all we're doing here is we're adding a retry. So basically if a tool is called, then we're good. That's valid. If it has meaningful text, we're good. But otherwise we do reprompt it. That's all we're doing here, right? We're just making sure that the LLM actually returned a valid response. So that's really all to worry about here and there. We're also creating this tool node. So this tool node basically just will try to invoke the tool, and it'll basically have a little, we're going to add a little thing to handle errors in the feedback. These are just like utility functions, so don't really worry too much about them. Now here's kind of the interesting bit. We're just going to build build the graph and it's going to look exactly like we show here so what we're going to do is we're going to add a node for our assistant right we're going to add a node for our tool node and that's kind of this piece and this piece that's our tool node and then we're going to add this conditional edge which is a tools condition, which is all it's going to be is this piece. It's basically going to take the result from the LM, is a tool called, if yes, go to the tool node, if no, end. And we can implement that right here. So this tools condition, that's all it's going to do. It's basically going to return either a tools invoked or end. And then we go from tools back to the assistant. Now let's run all this and we can see what's nice about LandGraph is we actually, it'll automatically lay this out as a graph for us. We can visualize it here. So what's going to happen is we're going to start, we're going to invoke our assistant. Our assistant will, in some cases, ask to use a tool. It'll then go to the tool node, the tool will be invoked,oked that'll return to the assistant and that will continue until there's natural language response and then we'll end that's it nice and easy so let's actually test this out um and i'm gonna go ahead let's ask a super simple question so let's look at what we're i've kind of two questions was magic function three and was the weather and sf let's ask question the first question What is magic function three and what is the weather NSF? Let's ask the first question. What's magic function three? Boom. So we're going to run this now. Now I'd like to go over to Langsmith and look at the result here. So let's actually just walk through this. This basically allows us to say we basically started. We went to our assistant. And these are the functions available to our assistant. So that's kind of, you know, we gave it magic function. We gave it web search. You know, here's the prompt. What's magic function three? And what we get as an output is, again, the function to use and the payload to pass to the function. So again, remember, this is kind of always a little bit of a confusing thing. An LLM can't magically call functions. An LLM is type string to string. It can return strings and it ingest strings. So that's fine. All it's going to return in this particular case is just the payload to run the function as well as the function name. But that's it. That's all the LLM is responsible for. Then what we need to do is we have this tools node, see that's here, that will then invoke our function. And so you can see the input is just the argument. The output is 3 plus 2, 5. Great. Now this goes back to our LLM. And then our LLM just simply sees, OK, it sees this tool message that the function was called. Here's the output of 5. And it returns natural language. The result of magic function is 5. And then we end. That's it. Nice and simple. And we can see that also kind of laid out here. Here's our human message. This is the AI message. So basically the AI makes the decision to invoke the tool and it gives you the input payload. Then here's the output tool message saying I ran the tool. Here's the output that LLM gets that back and basically gives you natural language. And then based upon our condition here, this tool's condition, if it's natural language, And then based upon our condition here, this tools condition, if it's natural language, it ends. If it's a tool invocation, it goes back to the tool node, right? So that goes to here. So in this regular case, it went back to the assistant and now it's a natural language response, which means we just end. That's it. So that's kind of a nice and simple example. Now, if we go, let's try something slightly more complicated. Let's try our other tools. Let's try, what's the weather in SF right now? We're going to try to run that. Cool, we can actually see that it's going to call our web search endpoint. That's great. It gets this kind of raw tool message back from the endpoint, and then the AI will synthesize that into, you know, the weather is 60 degrees right now with mist. Okay. So that's really it. This explains how you can lay out arbitrary agents with LLAMA3, open source LLM. We use chat grok to do that. Grok has been adapted for tool use, and that's the kind of main important thing you need to recognize that you need an LLM that actually has tool use enabled via prompting or fine-tuning or otherwise. And what you can see is, if we kind of go back to the diagram, what we've done here is we're using LineGraph to kind of orchestrate this process, and what's going to happen is you take a question in, our LLM makes the decision based on the question to invoke a tool, and then this conditional edge will determine, hey, if a tool is kind of invoked, then go to the tool node and actually execute the tool. The tool is executed. You get a tool message back with the tool output. Send that back to the LLM. LLM reasons again. And it could make a decision to call another tool, but in our particular case, in both cases, the tool message output was returned to the LLM. The LLM then responds in natural language. Here is the solution, and because of that, we end, and that's it. That's kind of how to build an agent from scratch using an open source LLM, LLM3, with a land graph to orchestrate it hopefully um from kind of kind of very simple components and first principles and again the key thing here really is the ability or the ability for an llm to reliably invoke tools so we talked through the case of adding two tools magic function and web search to our agent now let's say we want to make this a little bit more complicated and try some additional tools so replicate Replicate as a service allows you to access many different models, which is really convenient. And I'm going to go ahead and use it to augment Lama3 with a few multimodal capabilities. So all I've done is I've set my Replicate API key. So I've actually already done that. I've import Replicate. And I'm going to use a few different things here. So I'm going to do a text to cert, text to image tool, which is going to call this particular model, which is basically an OpenDolly model, which will go from text to image. I'm going to create, again, another tool, image to text, in this case, take an image in. It'll use a version of Lava to then produce text from the image, and text to speech. This is another option. So really all you need to do here is very simply just again, use this tool decorator with a function definition that invokes the model of choice. So now the question is how do we add these as tools to our agent? So again, it's kind of like before all we need to do is just update our tools list to include some of our new functions here. That's it. Pretty simple. Now that tools list is already bound to our agent here. So let's just go ahead and kind of rerun everything just to make sure this all works. And all I'm going to do here is just update my question list to include a few new questions that are related to my new tools and let's go ahead and try one so let's say i want to try um my index two questions so questions two and this is gonna be my question related to um image to uh this is gonna be text to image so let's basically say i'll kick this off and i'll go back and show you um so this is going to basically uh, hopefully invoke the text image tool based on this prompt, a yellow puppy running free with wild flowers in the mountains behind. So that's our prompt. We're going to pass it to our text image tool. And it looks like that has been called correctly, so that's great. Now we can also go over to Langsmith. I can check my projects here. Cool. Here's my agent. Here it is running. So we can also look at the trace to confirm that everything's working. So cool. So it looks like it is calling text image tool, so that's fantastic. That's running right now. Great. Our tool ran. Now we can check our image here. Look at that. Very nice. Again, this is just showing you the ability to create agents that have many different types of tools. Again, previously we only had covered two very simple tools, a magic function, web search, but we can actually do pretty interesting things. This actually shows how you can take replicate for example and basically invoke many different llms hosted by replicate or you know not just llms but different types of models so this is a text image model image of text and so forth text of speech basically to augment llama 3 and give it multimodal capabilities so in any case it's a really nice kind of illustration of the fact that capabilities. So in any case, it's a really nice kind of illustration of the fact that agents are very general and tools can be composed of many different kinds of things. In this particular case, different models through Replicate, which we can attach to LLAMA3 to augment its capabilities. Thanks.
Building open source LLM agents with Llama 3
Agents combine tool use, memory, and planning to build systems that are capable of short- or long-term autonomous tasks. Here, we show how to build agents from scratch, using Llama 3 with tool calling (via Groq) and LangGraph. Check out our Llama 3 recipes here!

https://github.com/meta-llama/llama-recipes/tree/main/recipes/use_cases/agents/langchain