 Good to be here. Good to see a fair number of people. It's early, so I wasn't sure if anyone would come, but thank you for coming. One quick note, I just put my slides on Twitter. I wasn't sure the best way to access everyone. I'm at rlancemartin. If there's another way I can get everyone's slides, then, yeah, I see some people opening. So the slides will link to a few different, it will link to a collab and other notebooks I'm providing, so all the code will be available for you. Okay, good, I see people finding it, that's fantastic. Colab and other notebooks I'm providing so all the code will be available for you Okay good as I see people finding it that's fantastic Maybe if there's a better way let me know but figure this is somewhat easy Well, it's great to be here. We have a bit of time So I think the format is I'll like lay out some slides to kind of set the preliminaries and like kind of give the big picture And then there'll be a bunch of time where I can just walk around talk to people and because we have I guess three hours so I think the idea of this will be a hands-on workshop I provide a bunch of starter code and also one of my slides shows it will be kind of a choose your own adventure format so why don't I kick it off I think maybe one or two people still coming in but so the the theme here is building and testing reliable agents and let me go to slideshow mode here and maybe I'll just kind of start with like the very basics you know LLM applications follow a general control flow of some sort you start usually to user input there's some set of steps and then you end and you've heard a lot about chains, you know, when you build applications, oftentimes we talk about this idea of chains. And chain is just basically, you know, it is some control flow set by the developer. Again, you start, proceed through some steps and you end. So retrieval augmented generation is a super popular application many, some of you may be familiar with, basically refers to retrieving documents from an index and passing them to an LLM. This is a good example of a chain. It's a control flow set by a user. The questions you know provided, vector store retrieves them, pass to an LLM, LLM produces an answer. So this is kind of a classic chain. Now when you get into agents there's a lot of different confusing interpretations. What is an agent? Here's the way I might think about it, which is just a really simple kind of framing is agent is just when the control flow set by an LLM. And so you can imagine we talked about this process of you start your app, step one, step two. In this case, I have an LLM in there. An LLM looks at the output of step one, makes the decision, do I go back and do I proceed? So that's like the simple way to think about an agent. So again, chains, developer defined control flow, I set it ahead of time, I follow some set of steps every time. An agent, an LLM kind of determines the control flow, an LLM makes a decision about where to go inside my application. That's one simple way to think about it. Now you hear about function calling a lot, and this is kind of a confusing topic, so I want to talk through it kind of carefully. Agents typically use function calling to kind of determine what step to go to. So usually the way this works is what you do is you basically give the LLM awareness of some number of tools or steps that it can take So in my little example here I define this tool and this little decorators a line chain thing But the point is I have some step it's some function I'm defining it as a tool and I'm what I'm binding it to the LLM So then LLM has awareness of this tool and here's the key point When it sees an input like what is the output of step two it actually produces the payload needed to run that tool now this is often confusing remember LMS are just string to string they don't have the ability to magically call some function what they can do is produce the payload or arguments and needed to run that function and the function name so really think about tool calling or function calling is just an LLM producing a structured output. Still, you know, obviously a string, but it's a structured output that then can be used to call a tool. So that's all the function calling is. And you might have heard of React agents. So the way to think about this is it's just basically binding some set of tools to my LLM. And again, we talked about tool calling. So LLM makes decisions about what step or what tool to use and you have some node that will call that tool so LLM says okay run step one I have some node that runs step one and passes the output of step one back to my agent react typically stands for like basically action so the LM choose the action a tool is a tool is run it observes the output that's what goes back to the agent, observe that tool response, thinks about what to do next, maybe runs another tool, and this runs in a loop until you end, and usually the end condition is the LLM just outputs a string response, not a tool call. So this is the way to think about a classic React agent. And it's really flexible, that's the nice thing about a react agent so basically it can implement many different control flows it can do step one only step two one two two one that's the beauty of these open-ended style react agents and these have a lot of promise these kind of flexible tool calling agents were really hyped last year they're still really hyped it's really exciting because they're flexible they're open-ended you can give them a task give them some tools and can just execute arbitrary control flows given those tools to solve open-ended problems the catch is and this is kind of the crux of what we're getting to with this workshop is they do have poor reliability or they can so you can get caught if you've played with agents so as you can see they kind of get caught on one step and they keep calling the same tool and you know really this is often caused by l and non-determinism lms are not deterministic and also errors in tool calling so tool calling is kind of tricky if you think about it lm has to basically pick the right tool given the input and has to pick the right payload so it has to produce the right inputs need to run the tool and these both can break so here's a good example the tool I'm passing is step 2 and the LLM is saying the tool name to run in step 3 so that's obviously wrong or I'm passing what is step 2 of the input 3 and the LM says okay pass 4 so these both these errors can happen tool calling is a tricky thing and it's exacerbated if you pass an LLM five tools ten tools it actually gets worse if you have very long dialogues it gets worse and so this idea of open-ended tool calling agents is really promising it's really exciting but it's really challenging because of these issues that we were mentioning here so this is kind of the build-up here like so can we envision something in the middle? So again, we talked about chains. They are not flexible, but they're very reliable. This chain will always run step one, two in order. We talked about like React agents on the other extreme. They're extremely flexible. They can run any sequence of tool calls. You know, can run step one, two, step one only, two only, two, one. But they do have reliability issues. So can we imagine something in the middle that's both flexible and reliable? So here's kind of the setup, and it's kind of the intuition. A lot of times in many applications, you have some idea of what you want the thing to do every time. So some parts of the application may be fixed. Like the developer can set, OK, I always want to run step one, and I want to end with step two. And you can inject an LLM in certain places that you want there to be some kind of branching or kind of optionality in the control flow Okay, so this is the motivation for what we call lang graph so lang graph is basically a library from the language team that can be used to express control flows as graphs and It is a very general tool. And I put out a bunch of videos on it. And we're going to use it today. And by the end of this, you will all have an agent that runs reliably using line graph, hopefully. And we'll see. So you should test me on that. If things don't work for you, then we'll work it out. But the idea is kind of this. This graph has some set of nodes and edges. So nodes you can think about are basically, well, maybe I should start with this. This graph has something called state. So it's like short-term memory that lives across the lifetime of this graph that contains things you want to operate on. Nodes modify the state in some way. So basically each node can like call a tool and can modify the state edges just make decisions about what node to go do next okay so you basically have this idea of memory and this is the same as common agents right majors are characterized by having tool calling and in short-term memory as well as planning those same things are present in line graph memory is this state that lives across your graph tools exist within within your nodes, and planning, basically, you can incorporate LLM dictated decision making in the edges of your graph. So like why is this interesting and where has this been cropping it? We've actually been seeing this theme crop up a lot of places. So there's a really interesting paper. There's actually a few I really like. This one's called corrective rag. And the idea is pretty simple. Like with a naive rag pipeline, you're doing retrieval, you're taking retrieved docs and generating your answer. Corrective rag is doing one step more, saying why don't we reflect on the docs we retrieved and ask are they actually relevant? You can have lots of issues with retrieval, you can reflect on the documents, see if they're relevant, if they're not relevant you can do different things, you can kick out and do a web search. So it makes your application a lot more dynamic to poor quality retrieval So this is one of the first videos I put on land graph back in February It was very popular and I basically showed you can build corrective rag inside land graph and it's super simple This is what the graph looks like I do retrieval I grade my documents and We're gonna actually we're gonna do this today and I have a bunch of code for you that does exactly this So we're gonna go way in detail on this one but this is kind of set up and I showed this working I showed it works locally with Olama using in fact at that time it was Mistral 7b and it works really well so this is like one simple illustration how you can use Lang graph to build kind of a self reflective or corrective rag application now another cool paper was called self rag, which actually looked at the generation. So basically we're all familiar with the idea of hallucinations, it's a real problem. Instead of just allowing hallucinations to propagate to the user, you can actually reflect on the answer relative to the documents and catch hallucinations. If there's hallucinations, you can basically do different things and they propose a few ideas here. implemented this and this is a this is actually our most popular video of all time so this was showing Lang graph and Lama 3 implementing three different things corrective rag which we just talked about the self rag thing of hallucination checking and this adaptive rag thing so I can kind of walk through it this all runs in Lang graph locally and I have the notebook here if you want to test that today you definitely could so that's the point is reliable enough to run this whole thing locally so what's happening here is I take a question I route it either to my index or to web search I then retrieve documents I grade them for relevance if any are not relevant I kick out and do a web search to supplement my retrieval if they're relevant I generate my answer I check it for hallucinations and then I finally check it for answer relevance so basically does it have hallucinations relative to my documents and does it answer my question if all that passes I go I finish and return that to the user so this is kind of like a complex rag flow but with lang graph you can actually run this on your laptop it is reliable enough to run a laptop with lang graph and the intuition again is that you're constraining the control flow you're allowing the lm to make certain decisions but at very discrete points if you implement this as a rag agent this could be very open-ended and a lot of opportunities for breakage and so that's the real intuition here um now a final theme is karpathy kind of mentioned this idea of flow engineering related to this alpha codium paper a really nice paper on code generation and the intuition here is produce a code solution they tested this on a bunch of coding challenge abuse a code solution and check it against the number of unit tests auto-generated or pre-existing and Basically if it fails unit tests feedback those error cell and try, really simple idea. I implemented this in LangGraph, again, the code is here. And this works really well. So I basically, I share a blog post as well. I ran this on our internal coding. We have an internal application for RAG at LangChain. And we're actually working on implementing this right now in production because the performance is way better. And a common thing this can fix with code generation and code solutions is hallucinations and imports so we see that a lot with our RAG app so what I did was I very simply implemented a unit test for import checks just run that it significantly improves performance relative to without doing it and so we're actually working on implementing this in our internal rag system so super simple idea that can really improve code generation so you know if I kind of like back up what what did we talk about and we talked about chains they are not flexible which is fine in some cases but a lot of interesting newer papers with rag for example this idea of self reflection is really beneficial the ability to kind of self-correct applications can be really beneficial not beyond RAG as well for coding. So chains are very reliable but they're not flexible. Now if you go to the other end like a classic react agent is very flexible. It can implement any sequence control flows through your different tools but it does have reliability problems due to things we talked about non-determinism, tool calling errors and line graph kind of sits in the middle where you can actually implement these user-defined slash LLM gated control flows and they can actually be extremely reliable because of that constraint they are less flexible than a classic react agent so that is true so for very open-ended tasks I do agree maybe you is true. So for very open-ended tasks, I do agree, maybe you do need a very open-ended, more autonomous style React agent, but for a lot of applications that our customers are working on and seeing, these kinds of hybrid flows are sufficient and you gain a lot of reliability. And so we've talked to a lot of companies that have implemented LandGraph successfully for agentic flows for this reason. Because reliability is just incredibly important in a production setting. So this gets into, if you look at the slides, I have a few different notebooks. And what I show is, we talked about corrective rag. These notebooks show how to build corrective rag yourself. And I thought that's a fun starting application. It's a really popular one. It's super simple. There are not many dependencies. You can use your own web, whatever tool you want to use your web search. You can use other things as well. You should have a look at the notebooks, and I'll kind of walk around. And we're going to keep going. This is just like a placeholder here. But so if you want to test this locally, if you have a laptop this locally people laptop cable over anything's locally then we have a notebook to support that I use a llama I can talk a lot about that that's a really cool thing if you don't then I have two options for you so one is a collab so that's probably the easiest if there's issues let me know if I've tested it so if if you have a Google account, you can spin up a colab. All I need is a few API keys depending on what models you want to use. It's all kind of there. You can set those accordingly. And I also have a notebook. So this just kind of is like a kind of a gives you a roadmap of the different things you can try today. Since this is a workshop format. And I'll just be walking around and we'll do questions for a while. But I want to talk about the second half of this of this you know story so one of the things we're seeing a lot I think you're gonna hear a lot at this conference is the challenge of testing and evaluation and this is a real pain point like for example how do I actually know that my land graph agent is more reliable than the react agent how do I know what LLM to use how do I actually know that my LangGraph agent is more reliable than the React agent? How do I know what LLM to use? How do I know what prompt to use? So testing agents is, testing in general is really hard and agents in particular is challenging. So there's kind of three types of testing loops I like to think about. One is this in-app error correction. And that's actually what we just talked about. So LangGraph agents are really good for that. So basically, in-app error handling, where you can catch and fix errors is really useful for code generation, for RAG, we just talked about that. So that's like placeholder one. Now we get into this idea of pre-production testing. And then finally production monitoring, and I want to introduce a few ideas on the latter two. So we just talked through this here we're going to build corrective rag a few different ways and I just showed the choose your own adventure stuff and so this is just kind of reiterating that but I want to show you some other things so Langsmith is a tool from the Langchain team that supports testing and evaluation as well as monitoring and tracing and so we've seen a lot of interest in this and it's quite popular. It is really useful for doing these types of testing and evaluations. So the key idea behind Langsmith and the notebooks actually have this so this is totally optional if you just want to build an agent that's totally fine. If you want to also test it you can you can use Langsmith you don't have to of course but I have it all set up to use Langsmith if you want. It's free to use, of course. And so the idea is there's kind of four components that I like to think about when it comes to testing slash evaluation. You have some data sets. That's some set of examples you want to test on. So say you have a RAG app. That's like a set of ground truth question and answer pairs you've built. Like you're testing your system. You have question and answer pairs that you know are correct. Can your system produce those answers? How many will actually get right? You have your application, that's your agent, that's your rag app, that's your code app, whatever that is, that's your application. Now the thing that's often the trickiest, you have this evaluator thing. And the notebooks show you in detail, but this evaluator is something as simple as a user defined function that can implement a few different things. You can think about using an LLM to actually judge your output. So in that case, let's take rag as an example My application produces an answer. I have a ground truth answer You can actually have an LLM look at those two answers jointly and reason is it correct? And this is often very effective or require some prompt engineering I have some nice templates in the notebooks to show you But this thing is very populous idea of LLM as judge evaluators is very interesting a lot of people actually you'll probably hear about this week it's a really good theme it's still kind of in development but that's like one placeholder to keep in mind so for one option for time testing is this idea of using LLMs themselves the other is building your own heuristic evaluator so a custom evaluator of some sort and actually the notebooks that I share have both and So we're actually gonna the notebooks actually show how to evaluate an agent specifically and there's a few different things You can look at with an agent. So one is you go to that far right in blue the end end performance So our notebooks are basically gonna be a a RAG agent. The eval set has five questions. And I basically have a set of question-answer pairs. So basically I'm going to compare my agent answers to reference answers. And we'll walk through that in the notebook. But that's kind of one thing. I just want to introduce the idea. So one big idea is you can evaluate the end-to-end performance of your agent. Right? You don't care anything about what's happening inside the end-to-end performance. The other, which I actually like to look at a lot, is this thing on top. What are actually the tool calls that the agent executed? This is how you can actually test the agent's reasoning. So what you see often with agents is they can make some weird trajectory of tool calls that's highly inefficient but still gets the right answer. You don't get that if you only look at the answer. You say, oh, okay, it's got the right answer, but if you look at the trajectory, it's some crazy path, and so you want to actually look at both, like how efficient, how correct is the trajectory, and does's got the right answer. But if you look at the trajectory, it's some crazy path. You want to look at both. How efficient, how correct is the trajectory and does it get the right answer. The notebooks I share actually do both. This is actually an evaluation that I ran. This data set is public. This is on the agents that we just talked about. This is what you see when you open Langsmith. These are different experiment names. This is just saying like I've run three replicates of each experiment. And these are my aggregate scores. So this first score is basically the answer correctness thing. And the second score is like the tool use trajectory, or like does it use the right reasoning trace. And I can go through my experiments. So this top one is actually, this is kind of cool, this is actually my local agent running on my laptop with LangGraph, okay. It's a five question, you've got a small eval set, just a bunch of, it's some very small test examples, but basically my local agent does not, does fine. It does 60% in terms of the ultimate answer, so that's not amazing, but it does do very well in terms of the tool calling trajectory so it's very reliable in terms of reasoning it's an 8 billion parameter model so I basically I think the quality of its outputs are a little bit lower than you might see with larger models now fire function v2 is another option it's basically a fine-tuned llama 70 B from fireworks this one with Lang graph so this is actually showing this top actually gets up to 80% so very strong performance in terms of answers and a hundred percent again in terms of tool calling so the key observation here is the tool calling or reasoning is consistent whether using a local model or 70 billion parameter model with Lang graph so you get very high consistency in your true calling the answer quality degrades that's more an llm capacity problem but the reasoning of the agent is consistent so that's the key point now here's where it gets interesting fire function v2 again that's llama 70b this is with uh a react agent what you can see here is the answer quality is degraded but here's the interesting thing the tool calling trajectories are really bad and this again gets back to that problem with react agents they're open-ended they can choose arbitrary sequences of tool calls and you can deviate really quickly from your expected trajectory so that's the key intuition here now the final two are gpd-4-0s that's obviously a very flagship model it's maybe number two now relative again sonnet on the chatbot arena at least you know again answers ultimately are are strong the tool calling though even here is degraded so basically it follows some weird trajectories to get to its answers that are unexpected so what's the high level point here the high level point is lang graph allows you to significantly constrain the control flow of your app and get higher liability and if you look at these tool calling scores it's very kind of consistent going all the way down to local models. It follows the same sequence every time. React agents kind of go off the rails much more easily. The answer performance is really a function of the model capacity. So using an 8 billion parameter model locally, the answer quality is lower than a 70 billion. That's to be expected. But the reasoning of my app is consistent and strong, so that's the key thing that you kind of get with LangGraph, and this is all public. And hopefully some of you will actually, you know, implement this or reproduce this today. And this is just walking through those same insights I just mentioned. And then deployment, we're going to be talking later this week, we have an announcement related to deployment of LangGraph, so this is actually a very good setup. If you're playing with LangGraph, you enjoy working with it, we're gonna have some really nice options for deploying later this week. And so Harrison will be here on Thursday to give a keynote on that one. And if you've deployed, we also have some really nice tools and lengths to actually monitor deployment. And this is not as relevant for this workshop. It's something just just be aware of. I can talk about if you're interested. So maybe to close out, so this is a really nice write up. These guys are actually going to give a keynote later this week. It's Jason and company, Hamel and others. And they kind of made a really nice point that the model is not the moat. Like, elements are always changing. The moat is really the systems you build around your application That's what we talked about today. Like do you have an orchestration framework for example like Lang graph? Do you have an evaluation chassis like Lang Smith and again? You don't have to use Lang graph. You don't have to use Lang Smith for these things But this workshop will introduce these ideas to you and frankly I think it's important to understand the ideas rather than the Implementation whether or not use Lang graph whether or not you use Langsmith I think understanding these these principles is still helpful but you know an evaluation chassis guard rails data fly well these are like the components that give you the ability to improve your app over time that's the that's really the big idea that's the goal and I think you'll hear more on that later this week the goal here is how are you measuring improvement of your app and ensuring it always gets better. That's what we're actually trying to achieve here and that's kind of what evaluation is giving you. And yeah this this is kind of my last slide then maybe we can just move into maybe some Q&A. I can actually show the notebooks themselves if you want to walk through them together. I mean I'll just do that I'll let you guys kind of hack on them in parallel as I walk through them and then I can just go walk around and talk to people something like that so you know the three types of feedback loops you have this design phase feedback something like line graph in app error handling that's kind of step one we talked about cool examples there for coding for rag a lot of nice papers really promising I'm very excited about anything you can do here with terms of kind of agentic self-correction self-reflection in your app itself pre-production testing we just talked through that building evaluation sets running evaluations testing for an agent like your tool use trajectory your answer quality all really interesting and important and And then in production phase, production monitoring, this gets into, we didn't talk about it too much, but basically this stuff. So basically you can have evaluators running on your app in production, looking at inputs, looking at outputs, tagging them accordingly, and then you can go back and look later. So that's kind of the setup here. I know that's probably a lot, and we went about half an hour. So if there's any questions, I can just open it up, and we can kind of talk through stuff. I can also start ripping through some of the notebooks just to kind of give you an overview of the code itself. But if there's any questions here, maybe I'm happy to take a few. I'll give you a minute to digest all that. Yeah? Is there a non-Twitter link to your slides yet? give you a minute to digest all that. Yeah? Is there a non-Twitter link to the slides yet? That's a good point. Let's see. Is there a non-Twitter link to the slides? Let's see if the conference organizers give me some. I don't know if I have an email list for everyone in here. Just post to the general channel. Is there a Slack? Yeah, if someone can put, I actually didn't know there's a an email list for everyone in here. Is there a Slack? I actually didn't know there's a Slack, so that's very helpful. If there's a Slack or an app for this conference, then please someone post. I appreciate that. I actually didn't know that. Thank you for that question. Yes, sir. So about testing and evaluation, does it really scale to predict the exact sequence of the agents? If it's smart enough dealing with a complex problem, it's hard to say exactly how it is achieved. Okay, this is a very good point. So I'm going to repeat the question now. How do you evaluate an agent's reasoning trajectory if it's a large and open-ended problem that can be solved in many different ways? In that particular case, you are right. It is hard to enumerate a specific trajectory that is actually reasonable. For really open-ended long running type problems Trajectory evaluation may not be appropriate It's one one thing I would think about is can you define a few canonical trajectories of tool use through your application so It depends on this that if it's a very long-running agent I think it's probably infeasible if it is a shorter run agent where it's like you expect something in the order of maybe five to ten steps you can probably enumerate some set of reasonable trajectories and basically check does it follow any of these. You can also do things so you can do things like this you can do things like check for the repeat of certain tool calls. You can be very flexible at this and it's kind of open and few so you can look for like is it repeating certain tool calls. can be very flexible at this and it's kind of open and you can look for like is it repeating certain tool calls you can look at recall like is it for sure calling this tool or not so you can actually be very flexible actually the way we set up Lang Lang Smith evaluator for this it's just a simple function that you can define yourself so it's a very good point you can be in credit you can be arbitrarily creative about how you evaluate that but I would say for very long running, you're right. You can't really articulate step one, two, and three. But I would then think about more like evaluating. Is it repeating steps? Can you evaluate for clearly aberrant behaviors, excessive number of tool use repeats, excessive number of overall tool calls, so kind of like guardrails related to like kind of clear aberrant behavior? It's very short-term you can actually enumerate the trajectory specifically that's a good question so in the code you can see we actually lay out a custom function we can define that yourself so that's a that's a very good point though yeah yep Can we transition the case to Human in the Loop? Yeah, Human in the Loop is a good one. The workshop notebooks I share do not have that. But LangGraph does have some good support for Human in the Loop. And I can share with you some notebooks that showcase that. Also, what we have shipping on Thursday has very good support for Human in the Loop. So I will share some notebooks with you for that and wait for Thursday for even more there. So if we're building a rack like application, right, pre-production, we can do this testing framework using a known set of this is a question, there's a deeper answer to the end, and I will present that. What I've been struggling with the drill is the right way to approach that in production where we don't know what the right answer is. And I'm wondering if you. Yeah. So the question was, this is actually a really good one. For RAG, I'm just going to go to our docs because I actually wrote a doc on this recently. So you can still see my slides. For RAG, in a pre-production setting, it's easy to define, or not even easy, but you can define a set of question-answer pairs and evaluate them. When you're in production, though, how do you evaluate your app because you don't have a ground truth answer? So what are other things you can actually evaluate for rag app that don't require a reference yeah so there is a conceptual guide that I will share so this is actually a rag section I've kind of a nice overview of this there's actually a few different things you can evaluate for rag that don't require a reference that are very useful yeah so it's this right here so this is like a typical rag flow so I have a question I retrieve documents I pass them to an LLM I get an answer right what we just talked about and we showed is comparing your answer to some reference answer now this, this is, to be honest, pretty hard to do. You have to build an eval set of question answer pairs. Very important to do, but it's not easy. So what else can you do? So some of these we've seen that are really easy and actually pretty popular, there's three different types of grading you can do that don't require a reference, that are like internal checks you can run. I mean, you can run them online. So one is retrieval grading so basically looking at the doc your retrieve documents relative to your question so like an internal consult consistency check so this is actually a great check to run and actually the corrective rag paper that or the corrective rag thing that is in the cookbooks that I share here does this so you can play with the prompt and all that but basically this is just checking the consistency you retrieve docs relative to your question you can do that and we have some really good prompts for that another one I like is just comparing your answer to your question have an LLM look at here's my answer here's the question is this sane are they related and this is a really nice check just for like you know of course you don't have a reference answer but like you can still sanity check and say does this deviate significantly from what the questioner is asking the other this is a great one is hallucination and this is this is super intuitive compare your answer to the retrieve documents so if the LLM went off the rails and didn't ground the answer properly and you hallucinated you can catch that really easily and so I need to get on this slack because I want to share this link with you I'll figure that out I'll find you but this is in our Langsmith Docs if you search Langsmith valuation concepts we have actually a bunch of videos that showcase how to do this and I have a bunch of code as well so but those are three things you can do that don't require a reference and we do run those in as online evaluation with our application so yeah yep yeah unit testing so um yeah okay so do we have any thoughts on unit testing? So Langsmith supports PyTests as unit tests, but basically, it depends what you mean by unit tests. Typically, like conventional software engineering unit tests are very effectively done in things like PyTests. There's a lot of frameworks for that. What I like to think about in unit testing with respect to LLM apps is kind of like what we show in this code generation example. Here we use some really simple unit tests just for imports and code execution. Simple unit tests like this can actually run in your app itself. So basically one place you can think about putting unit tests are actually in your app itself within LangGraph for self for kind of in-app error handling or self correction. So that's like one place for unit tests that's kind of interesting and new with LM applications. They can live inside your app itself. Another good one for unit tests within app is if you're doing structured output anywhere in your application, which is like a really common thing people like, confirm the schema is correct. That's another good use case for unit tests in your application which is like a really common thing people like confirm the schema is correct that's another good use case for unit tests within your application those also both of those things could also be done independently like in CI outside of your application so we are going to have more integration support for CI with Langsmith soon I will check with the team on that. But I think the interesting idea for unit tests with LLM applications is this idea of inline within your app itself, because LLMs are so good at self-correcting. If you run unit tests in your application, they can often catch the error and then correct themselves. And unit tests are fast and cheap to run, so it's actually a really nice piece of alpha that, in fact, that's exactly what Karpathy was mentioning here, that running unit tests in line with your application is actually really quite nice and produce significant improvement in performance and alpha codiums cool yep I forgot to say that you can use a small area of. OK. Yeah, yeah. So what the question was, if I wanted to do some of this in-app error correction stuff, so let's take this example, the corrective rag thing. If I actually want this to run in my application, it obviously needs to be super fast. So that's actually what we've seen. The tricks we've seen here are basically use very fast smaller LMS so you mentioned for example even the ability to fine-tune that's actually a good idea if you have a judging task that is very consistent it's a very good use case for fine-tuning actually fine-tune a small low capacity extremely fast and effectively very cheap to deploy model that's a very good idea we've seen people do that also use very simple grading criteria don't have some kind of arbitrary scale from zero to five with high cognitive load the LM has to think about yes no very simple binary grading even for some of the stuff you can even be old-school and fine-tune a classifier but basically really simple lightweight fast LLM as judge style classifier wouldn't be an LLM necessarily but but basically very simple fast test for in-app anything like kind of in runtime you will need or want another cool use case for this or another like interesting option for this is grok is very very, kind of with their LPU stuff. And they actually would be a very interesting option. We've done some work on that with Grok, basically, for any of these kind of in-app LMS judge error correction things using something like Grok, which is extremely fast. But it's a very good insight. Fine tuning your own model is actually a really good idea. We've seen people do that for these types of anything with LandGraph and in-app error correction cool yep yeah we have some good cookbooks talking about multi-agent which I, again, will need to find a way to share with you. Yep. So we have this. If you go to LangGraph GitHub, examples multi-agent, there's a few different notebooks here that are worth checking out. Yep. Is this LangGraph allowing you to find cyclical graphs? Yep. Yeah. Yeah. Yeah. So actually, LandGraph is specifically designed for cycles. So some of the examples, like what we're showing today is only a branch, so it's a simpler graph. But for example, the React agent we will show today is a cycle so it's basically gonna gonna continue in a loop just like this and what you do is you set a recursion limit in your line graph config so you basically tell it to only proceed for some number of cycles this is default set for you I believe it's like 20 or something like that but But that's what you're going to want to do. Yep. So the question was about timing of the responses. So do you mean like if you're implementing some kind of self-correction, how long that takes? Well, that gets back to the question that that this gentleman asked it depends a lot on the LM you choose to use for your judging and its latency so that's kind of where if you're so maybe there's two sides of this one side of it is choosing an LLM that's very very fast and that's very important to do could be something like grok could be a fine-tuned like deployment that you do yourself could be a gpd3.5 so that's like one side of it the other side of it is um how do you actually kind of monitor measure that and so again langsmith actually does have very good support for tracing and observability and we do have timings all in fact I can go ahead and show you very quickly if you want to see So this is my Langsmith dashboard. I'll zoom in a little bit. These are my experiments Now if I if I zoom in here, I can open up one The Wi-Fi is a little bit slow. These are my replicates I can open up one of my traces and what I can see here is over here I get the timing. So this is the timing of the entire graph and I can go through my steps This is like the retrieval is really fast. That's good. You know less than a second Okay, now here's what's interesting my grading in this particular case is like four seconds So that's you know not acceptable in a production setting most likely But again, this is just like a this is a test case in fact I'm using what I'm using to grade here of GPT 4.0 so it is You know there's ways you could speed this up by using different models or different prompts or grading everything in bulk So there's a lot of ideas. I should grade each document independently. Oh, actually, you know what this is using chat fireworks So anyway, but you can look at your timings in lengths, but that's a really nice way I'd like to do it to kind of see, to kind of monitor the timing of my applications. Yep. The case you showed, like, from scratch, but when the agent is running from the beginning and trying to accomplish the task, but in reality, usually people have some context or history that's being passed to an agent. Do you have any suggestions of how the task is going through? Because, say, they don't have to do a function call because that's already in the history versus't have to do a function call because that's already in the history versus like. Let me try. I got the question. So the question was with agents, you typically pass them a message history. That is absolutely true. And in fact, the React agent that we we implement here like I can even open up one of the traces we can look at it together so so here's a react agent with GPT 4.0 here's one of my traces let's open it up and actually see what's going on here so what happens is first my assistant is right here. So this is OpenAI. Will, you know, here's the system prompt, right? So your helpful assistant, you're answering questions. Here's the human question, right? So again, this is the start of our message history, okay? So what the, and also these are the tools that the LLM has. And this is pretty cool. We can see this is the one it called. So what happened is our LLM, it looked at, here's our system prompt, here's the human instruction, and it says, OK, I'm going to retrieve documents. Great. So then it goes, and this is a tool call. It goes and retrieves the documents. Now that goes, so you look here, we can actually open up the retriever. Here's our documents. Cool. Now it goes back to the assistant. So this is a looping thing actually open up the retriever. What do we get? Here's our documents cool Now I go it goes back to the assistant so back to this is a looping thing It started our assistant it made the tool call the tool ran we got documents now we get them back now Let's go back to our LLM. So now our LLM. This is pretty cool, right? Here's the message history like you were saying instructions question document retrieval the documents that are retrieved are right here. And then now the LLM says, okay, I want to grade them. It calls the grader tool. And this is its reasoning and this is its grade. So anyway, you are right that as this goes through, you basically accumulate a message history. And the LLM will use the message history and most recent instruction to reason about what tool to call next that's exactly how it works I think I answered the question is is there anything that isn't clear about that so it is true like let's look at an example right so in this particular case right the LLM sees the retrieved documents from the tool and then it makes the decisions is okay I have this tool response it's retrieved documents right the LLM sees the retrieve documents from the tool and then it makes a decision it says okay I have this tool response it's retrieved documents what should I do next and it says okay well why don't I go and grade them and it calls a grade tool so it looks at the message history and reasons about what tool it'll call and that's exactly how these react style agents work and the whole issue is that's kind of a noisy process like it can look at that whole trajectory it can get confused, it can call the wrong tool, then it's on the wrong track, and that's exactly why these more open-ended tool calling agents fail. Say like in the second follow-up question to the agent, that technically it should follow similar paths in calling, right?... Okay, right, so I think that the question is, well, let's say there's a multi-turn conversation where I can, the user can go ahead and ask a second question, of course, and that whole kind of message history will be propagated. yes that is that is a common pattern and that let's see I mean what's the question on that though like it it could use context from its initial trajectory to answer the second question for sure it'll probably look at that jointly when it's deciding what tool to call so for example if it receives a question and in its message history it sees the context need to answer the question the agent could probably decide okay I don't need to retrieve documents again I have the documents need to answer the question I'll answer that question directly so it is true that a multi-term conversation the agent can look at its message history to inform what to do next. That's definitely true Here I don't consider evaluation of multi-turn conversations, but that is it's a good topic actually I don't quite have a tutorial for that yet, but I Could think about putting that together. Yeah, that's a good point I'll make a note of that actually so multi turn is a good one cool okay yep yep because they reduce the hallucinations you can actually run within the app and so that's what have you experienced with the writing tests and then you're going to do that for an open and the full gen tests? So the question is, I believe, have I tested like the ability to kind of online auto generate unit tests yeah okay so that's a big topic so basically the Alpha Kodium paper that Karpathy references here does that I have not tested that because it does ramp up the complexity because then you're relying on and I don't think I mean that would be aggressive for a production setting because basically be relying on an LLM to auto generate you unit tests testing against things that are auto generated there's a lot of opportunity for error there I think it's interesting particularly in terms of like offline challenges like this but in terms of like a production application that feels pretty hard and risky, but it's an interesting theme. The thing I've tested more on I found to be very effective is super simple, crisp, lightweight, free, effectively unit tests. Like, again, the good use case I found was our, so Langchain, we have an internal rag application called chat Langchain and it indexes our documents and provides QA it occasionally hallucinates the imports and that's a really bad experience right if you take a code block copy it from this app and then it this import doesn't exist it's like what the hell you know that's really annoying so I incorporate a really simple check where I have a unit test that just, it does a function call where it extracts the imports from the code block from the answer and it tests the imports in isolation. If they don't exist, there's an error. I feed it back to LM and say, look, this isn't a real import. Try again. And I can do, you can do other tricks like then context stuff, relevant documents or something like that. Anyway, you can handle that differently. But that little alpha significantly improved our performance. So I kind of like simple, lightweight, free unit tests. The idea of online-generated unit tests is interesting but opens up a lot more surface area for errors. The idea is that, you know, if you think about how do you just like they oftentimes seem easier to write than the code and without a better structure. So we use that for, like, not reviewing the outputs of some things, but have the and then we do see the other simulation and then we get right to them going straighter and right to them before you actually do a better implementation. Yeah. Yeah, so the follow-up there was, let's see, trying to write the test before the app is implemented. I see, yes. So they also do that. So basically for each question, they do have they so the Alpha Kodium work references existing unit tests for a given question as well as auto generates so you are right that would be interesting to have a bunch of pre generated unit tests that you know are good for certain questions and to run them absolutely hard to do in a production setting with an open-ended input but potentially very useful in well okay even a production said you could maybe have some battery of unit tests and based upon the question type pull related unit tests that you know are gonna be relevant it's a good idea yeah it's a good idea for sure so basically using some battery of dynamically chosen pre-existing unit tests based on the question type or the documentation whatever documentation they're asking about that's a good idea yeah he was saying so for larger projects you can also use that to test for aggressions yes that is that's definitely a good idea and this paper it does incorporate that idea as well as this auto-generated unit test thing Which is a little bit more aggressive Cool yep for instance, LLM checking. There are some frameworks that have to do this with. It's added complexity. Yeah. It's added latency. I was wondering if you've seen those give more better results to justify the latency complexity or whether you've seen one checker, one reflection of this. Yep. So the question was, in some of these self-reflective applications, like let's say this one, the self-reflective applications like let's say this one the self-reflective rag Right. We're doing a few different checks here. We're checking documents. We're checking hallucinations. We're checking answer quality Instead of having some hard-coded single prompt To do that. Can you have another agent like kind of a checker or a greater agent that's a little bit more sophisticated I have and he mentioned a few frameworks I have not played with them I think it's interesting I think it's it's one of these things where it's really good for kind of academic papers it's very interesting it's really good it could be good for offline testing in an online or production setting or it's the latency and complexity is probably a bit much I think in a production setting it goes back to what I think this guy was referencing you probably would want something that's extremely fast lightweight and I would not think about like a multi-agent system in a production setting doing any kind of grading this whole idea of LLM as LLM graders look kind of a new idea so I think this idea of like more complex agent graders is interesting but we're kind of in the we're taking baby steps at this point especially in a production setting so I'd probably shy away from that for now particularly everything about production but first thing offline or experimentation it's probably interesting. Yeah. What's the best practice that you've seen so far for performing tasks or evaluations on those cycles? Because obviously you can end up with many, many different types of input and that sort of thing. And obviously LaneGraph provides some tracing there. But what about, hey, here's an interesting test case. Yeah, exactly. So actually the notebook shared here go into it a little bit. So the way I like to do it is, and I can even, why don't I just show you one of the notebooks. So basically, so at the bottom I have kind of all the different evaluations. So this goes back to a question that someone mentioned down here as well. So this is a, so there's a colab and then there's a notebook and they are both the same. So they have the same evaluation section. This is, this is ragag testing dot IPNB and there's also a colab which is it's the same notebook basically it's in the slides just make sure you got those links but but to your question so the way I did it and you can be very flexible this is basically I define the expected trajectory through the nodes in my graph and in this particular simple case the trajectories that I expect are basically retrieval grading web search generate or retrieval grade generate those are the two expected trajectories I want to take now in this case I don't do cycles if you did cycles you could just incorporate more steps here to say okay here's kind of the expected number of steps I want or expect to see through that cycle and I think someone mentioned here if you've a really open-ended challenging task then it may be hard to enumerate exactly but for a lot of like more production setting applications where these are not extremely long-running you can enumerate here's the steps I expect to take through the cycle and and the way I do the evaluator it's as simple as this there's not much code it's all in the notebooks but basically I compare the tool calls of the thing did to these trajectories that's it super simple so that's how I would think about it I keep it really simple I would basically try to enumerate here's the steps through the difference the cycle I wanted to take and yeah go ahead right yep that's it that's it that's it so it's actually really simple in in this is actually a good point in The land graph case for this custom agent. You'll see all I do is For every node and we need to get into the code if we want maybe people have already explored so Well, I'll just answer the question directly then we can back up and go through all the code if we want. But basically, each node in my graph, I just append this step name. So like retrieve, I just say retrieve documents. This is my generate node. I just append this thing generate answer. And I return that in my state as steps. So then I have this record of the steps I took in my graph that's it and I can go ahead and fetch that at eval time and compare it to what I expect that's all you're doing so that's how it work with like the custom line graph thing and now with the react agent actually it's a little easier because the react agent uses a message history so I can just go to my message history and that's exactly what I show here. You can strip out I guess I do it up above but basically I have this function in the notebook Let's see. Where is it? Yeah, it's fine tool calls react. So it's this little function that will Basically look at my message history and strip out all the tool calls so Yeah, it's a nice little nice little thing with a react agent it's really easy to get the tool calls out with Lang graph I just log them at every node and then at eval time I just can extract them yep Yep. So if you elaborate on the open-turn scenario, like there's context switching, where is it actually the cognitive ability that is a high-level count, you can't see it. And ultimately, it's going to treat, where is that cognitive ability? And then it's very valid point in the ground. Yeah. So the question was, with an agent in a multi-turn conversation setting, how does it know whether or not it has the answer? To a given question. Where to go, exactly. Yeah, that's right. So there's a couple different ways to kind of break this down so with these agents there's a few levels of instruction you give it first you give it an overall agent prompt so if I look at the notebook here we can go look at the react agent as an example of this. So the React agent is defined right here. This is kind of like the planning step. Here's my, like, naive prompt, okay? So your helpful assistant is tasked with answering tasks. Use the provided vector store to retrieve documents, grade them, and go on. Now let's take the case that's more complicated. Let's say I had two vector stores. So one thing I can do is I can explicitly put in the agent prompt, you have two vector stores, A, B. A has this, B has this. And then you're implicitly having the LLM giving it the ability to kind of reason out which one to use. Now this is where the second piece comes in. You also bind it to a set of tools. This is really where the second piece comes in you have to also you also bind it to a set of tools This is really where the decision-making comes in when you create this tool so here's retrieve documents, right this tool description is Captured by the agents the agent knows what's in this tool and this is really where that decision to use this Retriever tool versus another one would be made it'd be a combination of the prompt you give to the agent and or the tool description. So if you had two vector stores, you could basically say, like, retrieve documents one. This vector store contains information about x. Another one has information about y. Then the agent is deciding what tool to call based on that description and maybe based on its overall prompt. But to your point it's not easy so actually with custom agent this is with the react style agent with the land graph custom agent you can do it a little bit differently where I actually don't have it in this notebook but I have other cases where you can actually can build a router node and I mean I'll show you actually so this particular notebook so this one this self-rack it's in the slides if you open this up we did this with llama folks so this is actually a trick I really like if you go to if you go to lang graph rag agent local here see it's Wi-Fi is a little slow. What I define here is a very specific router at the start of my agent that decides where to send the query. And this is something I really like to do, because like we saw with the React thing, it has to kind of decide the right tool to use, which can be kind of noisy, versus right so here's my router right this is reliable not turn locally and what I do here is I run this at the start of my graph and I have the agent or yeah the agent explicitly take the question decide what to use and based on what decision it makes I can send it to either web search or in this case a vector store so to answer your question if I pull all the way back I Personally like to do explicit routing as a note at the start of my graph Because it's it's pretty clean and you can see in the flow of this overall graph this router runs first And it looks at my question and sends it to one of two places And this can be more complex you can send it to one of n places but as you wanted to here this is what like a custom line graph agent this is what I like to do if you're using a react agent it is then using a combination of the the tool definitions and the overall agent prompt to decide what tool to call but you can see it's more squishy because it has to call the right tool and as opposed to giving it a router and saying always run this router first it has to kind of make the right decision as to what tool to call based on the context question which is harder and that gets back to our overall story about these kind of Lang graph explicitly defined agents are more reliable because you can like lay out this routing step right up front and have it always execute that step before going forward. So this seems to be very deterministic. Everything's around. And what was exactly like, actually, a graph would say, do I have a history of the answer? Can I answer that? That's the thing that we did counted. Because then you have not only had the trees, or you all have to have the history. Can I answer the question? I'm just curious. How do you orchestrate that? OK, got it. So the question was, how do I incorporate the idea of routing with history? So here's what you do. It's actually, you know, kind of, it should be pretty straightforward. You can define this router node in your graph. And that router node, I'll actually go down to it. So basically, here's my graph and here's the the route question node, right? Yeah, actually in this particular case, it's it's an edge don't worry about those details Basically what you could do is you could have a node that takes in the state. Now that state could include your history. So what you could do is in that router prompt, you could really easily, here, include another placeholder variable for like your message history or something. And what you could say then is make a decision about where to go next based upon the question and based upon something in our history and so you actually would plum in your message history here and use it to jointly decide what to do next so that's actually really easily handled in line graph using a node and you can reference the history they can be passed you can pass into that node as state cool yep the notion of state yeah let's let's talk about state in a little bit more detail so let's actually go to the notebooks that we're working with here that I've shared so here's this ragged and testing notebook so if you go down to the custom land graph agent the way you do it is find the state yeah so here is what I call graph state so the graph state is basically something that lives across the lifetime of my graph I typically like to do something as simple as just a dictionary so basically this is a rag graph and here I'm basically going to find a number of attributes in my state that are relevant to what I want to do with rag a question A question, my answer generated, whether or not to run search, some documents, my step list. And basically, the idea here is that I define my state up front, and then at every node, I accept state as the input, and then I operate on it in some way, and right back out to state. So basically, what's happening is, I define state generally up front as like a dictionary or something like that. It's a placeholder for things I want to modify throughout my graph, throughout my agent. And every node just takes in state, does something and writes back out to state. That's really it. So basically it's a way you can think of it as a really simple mechanism to persist information across the lifetime of my agent and for this rag agent it's things that are really intuitive for rag it's like question it's documents and so let's take an example like okay here's a fun one so my great documents node right what I'm doing here is I'm taking in state and from my state it's just a right what I'm doing here is I'm taking in state and from my state It's just a dictionary. So I'm extracting my question. I'm extracting my documents And I'm appending my I'm appending a new step because I'm notifying. Hey, here's my new step and Basically, I'm doing some operation. I'm iterating through my documents. I'm grading each one the grades yes, I keep it if if the grade is no so yes no means like is it relevant or not basically so but yes it's relevant I keep it I put in the filter Docs list if it's not I've set the search flag the yes which means I'm gonna run web search because I want to supplement I have some docs that are irrelevant and I write back to state at the end. My filter docs, my question, search, and the steps. That's it. So state's a really convenient way to just pass information across my agent. And I like using a dictionary, just like nice and clean to manage, as opposed to a message history, which is like a little more confusing. Like in any node, if you use a message history, it's like a stack of messages so if you want the question you have to like it's like usually the first message you need to just index it's just kind of ugly I like using a dick which is like I just get the question out as a as a key in my dictionary cool let's see we're about an hour in. I can also let people just hack and walk around, talk to people, stay up here, whatever is best. And you keep asking questions if you want to. I think people are just working, doing their own thing now anyway. So I might ask one question just for fun. Is anyone interested in local agents? We didn't talk about that too much. It's a big theme. Yeah. So I shared a notebook for that. And by default, I am using Llama 3. You can absolutely test other models. So this is set up to test just Llama 3 with a Llama. Try other things. I have a M2 32 gig, so 8B runs fine for me. If you sling bigger, you could actually bump that up a little bit. So that can be a nice idea. Yeah, exactly. so that can be a nice idea yeah exactly 70 B is it's unfortunate that yeah I have I actually want a bigger machine so I can run that because I found for tool calling 8b is really at the edge of reliability So that's actually why you really can't run the react agent locally with an 8 billion model reliably you can run the land graph Agent very reliably because it doesn't actually need tool calling it only needs structured outputs you'll see in the notebook So that's actually a really nice thing But react agent won't run locally reliably at least Yep You say chunk documents, okay, yeah, so this is this question always comes up with rag. What is the typical chunk? Size yeah, if you ask this question always comes up with rag. What is the typical chunk? size Yeah, if you ask ten people you'll get ten answers it's notoriously ad hoc you know what to be honest I did a I Did kind of a talk on the future of rag with long context models I'm kind of a fan of Trying to keep chunk sizes as large as possible. Actually, I think this is a whole tangent, but I actually think one of the nicest tricks, let me see if I have a good visual for it. Basically, let me try to find something here. This is a whole separate talk. But basically, I think, yeah, this one, rag and long context. So this is a whole different thing. But yeah, this idea. So I think for rag, one of the ideas I like the most is decoupling. I'll explain this, I'll just say it and then I'll explain it. Decoupling what you actually index for retrieval from what you pass the LLM, because you have this weird tension, right? Smaller semantic-related chunks are good for retrieval relative to a question, right? But LLMs can process huge amounts of context at this point, you know, up to, say, a million tokens. So historically, what we would do is you would chunk really small, like try to get it as tight as possible, all these tricks, semantic chunking, a lot of things to really compress down to just compress and group semantic-related chunks of contexts, right? But the problem is then you're passing to the LLM very narrow chunks of information, which the problem is then you're passing it to the LLM very narrow chunks of information which has problems in recall so basically it's more likely you'll miss context necessary so it's good for retrieval but bad for answer generation so a nice trick is for retrieval use some chunking strategy whatever you want like make it you know small but you can actually use like a doc store to store the full document and what you can do is retrieve based on small chunks but then reference the full document and pass the full document to the LLM for actual generation time so you decouple the problem of like are you sure you're passing sufficient context to LLM itself now I also understand if you've massive documents It's gonna be waste one of those tokens to pass full documents through but There's some Pareto optimum here where I think In that being too strict with your indexing approach doesn't make sense anymore given that you can Process very large context in your LLM. So you want to avoid being overly restrictive with your chunk size So this is a nice way to get around it basically to summarize You can choose a chunking strategy whatever one you want But I like this idea of referencing full documents and then basically passing full documents to the LLM for the answer itself It gets around the problem of an overly aggressive chunking strategy that misses context needed to actually answer your question and I think with long context LMS getting cheaper and cheaper this is starting to make more sense I'm trying to think I had a whole kind of like slide on yeah this one is kind of like balancing system complexity and latency so it's kind of like on the left is like maybe the historical view like you'd the exact relevant chunk. You can get really complex chunking schemes, a lot of over-engineering, lower recall. Like if you're passing 100 token chunks to your LLM for the final answer, you might miss the exact part of the document that's necessary. Very sensitive to chunk size K, all these weird parameters, right? On the other extreme, just throw in everything, throw everything into context. Google, actually I think this week will probably announce some interesting stuff with context caching seems really cool maybe that actually could be a really good option for this but higher latency higher token usage can't audit retrieval security authentication like if you're passing 10 million tokens of context in for your answer generation. So something in the middle is what I'm advocating for. And I think this kind of document level decoupling, where basically you index and reference full documents and pass full documents to your LLM, is a nice trick. We've seen a lot of people use this pretty effectively so yeah Okay, that's interesting. So the question was, how can you evaluate the amount of context you are using? Okay. Yeah, like does it definitely use the whole state? I don't want to miss it. Okay, but I guess, so in a RAG context, context you have a question you have an answer. So you have a question you have an answer and you have some retrieved documents. So you can evaluate the question relative to your document. That's one way to get at this. Like how much of the document is relevant to your question? So that's maybe one approach. And actually the notebooks show a few prompts to kind of get at that and actually a good way to think about that is you can you can think about document precision and document recall and this is a little confusing maybe so I should explain it basically document recall is does a document contain the answer to my question anywhere so let's say you have a hundred page document on page 55 is my answer recalls one it's in there precision is the other side of that coin which is doesn't get any information not relevant to my question in that particular case huge amount of irrelevant information so recall be one precision be very low so that's one thing you can do you can actually look at your retrieve docs measure precision and recall that's like one thing I think I would probably like to do there. And that's probably the best way to get at this question of like, how much of my documents am I actually using? Now, with this approach, your recall will be high, your precision will be kind of low, and you would say, I don't care, it's fine. If I have a model that's super cheap to use large number of tokens, I okay maybe I'll frame it another way I care more about recall than precision I want to make sure I always I answer the question if I pass a little bit more context than necessary I'm okay with that versus if you're a precision gated system then you would say okay I'm gonna miss the answer sometimes I'm okay with X I never want to pass more context than necessary. I think a lot of people are moving towards higher recall because the LLMs are getting cheaper and they can process more context. So that's kind of how I might think about it. And I think this approach is a nice idea of indexing full documents or indexing chunks but then referencing full documents, passing full documents to your LM to actually generate answers. So yep. I was wondering if it seems like there's a couple of options for where exactly you inject the context into the conversation history. Like just for one example, you could stick something into the system and say, hey, this is your knowledge that you know. And then the user just asks a question and there's no prompting from the user. The other thing is you can completely modify what the quote unquote user is as in saying answer this question with this context and then it's in there. And then like a further idea on top of that, you ask a question and you ask a follow-up question. Maybe if you say, please elaborate, there's no need to actually retrieve anything, but you're just having to chat. But if you can chat, or you ask something else to be retrievable, do you have any observations as far as things that work well, or have to work worked well or continually appending things into the system or like adding another user like where do you stick that context how much of it history yep yeah okay so this is a great question the question was related to kind of like agentic rag and where to actually put documents. So let's walk through the cases. So case one is I have a fixed context for every question you're just going to ask. Let's say I have a rag bot against one particular document. And that document is always going to be referenced. So you make a very good point. In that case what I would do is let's go back to our like agent example you can put that in the system prompt itself like you said so and actually I'll mention something else about this it'd be like right here so here's your system prompt your agent you just plumb that whole document in you say every question you're going to reference this document you don't need retrie every question you're going to reference this document You don't need retriever system. You're done. That's a very nice case No, the retrieval complexity use context stuff your whole document The thing that Google is going to announce this week I believe this context caching seems really interesting for this because basically what they're saying is if I get it right I think Logan will speak to this, but, and I think they have a context when a million to 10 million tokens is huge, right? So basically you can take a large set of documents and stuff them into this model effectively. And they house them for you somehow. I think you've some minor data storage fee. But then for every inference call, they don't charge you for all those tokens that are cached it's pretty nice so basically here's the use case to your point exactly I have some set of documentation it's 10 million tokens that's a lot of pages hundreds of pages I have it cached with the model and every time that user asks a question I don't get charged you know 10 millions of tokens to process the answer they just cached for me really nice idea so that's your point of like the first thing that's like not quite in the system prompt it's in like the cache but that's the same idea so you have cached or system prompt fixed context that's like case one so case two is you have you want to dynamically retrieve so you don't you can't stuff your context maybe you have a few different vector stores like we were talking about here with routing in that case yeah you have context. Maybe you have a few different vectors like we were talking about here with routing. In that case, yeah, you have to use an index of some sort, maybe a router to choose which index to retrieve from. So that's kind of case two. I'm trying to remember the, oh, OK. So like in that particular case for follow-up questions, how do I kind of control whether I retrieve or not so that's the nice thing about? Either one of these agents it has some state so the state lives across lights on an agent So basically the agent and this actually gets exactly what this the other question was on Let's say I built my agent with I'll show you right here so let's say I have a router node at the start of my agent okay and that router has access to state what I can do is then given a question this could be let's say it's a multi-turn thing it's a second question in my in my conversation I have an appended state from the rest of my discussion here. The agent knows it returned an answer. So basically, when a new question comes in, you could pass the entire state back to that router. And the router could know, OK, here's the docs I've already retrieved. And it can basically then decide to answer directly because I already have the answer to the question. So that's a long way of saying you can use state, either message history or explicitly defined in your land graph agent to preserve docs you've retrieved already and then to just use them to answer a question without re-retrieving. So that's kind of what these graph, these rag agents can be really good at. That was kind of like storing that in short term memory and reasoning about, hey, do I need to re-retrieve or not? So that's exactly the intuition behind why these RAG agents can be pretty nice. Yeah? Yeah. Yeah. Yeah. Yeah Yeah Yeah Yeah, exactly. Okay. This is a really good. This is a really good discussion and whole debate. So the highest level framing of this is How do you want your model to learn? So one option is you can modify the weights of the model itself with something like fine tuning. Another is you can use what we call in-context learning through your prompt. So RAG is like a form of in-context learning. I'm basically giving it some documents. It's reasonable as documents producing answers, not touching the weights my model Fine-tuning be taking the knowledge I want to run rag on fine-tune your model and updating the weight so it has that knowledge there's a lot of debates on this and The and actually I think Hamel has a whole course on fine-tuning. Oh, yeah, do you have a Okay I just want to clarify. I don't mean fine tuning information. I just mean fine tuning so that you have the idea. If you can fine tune it so that it knows, oh, I should have always retrieved this information from my system. And use that when answering questions. Before you can find the model, you really use whatever extra context the user is providing. So just as a simple example of that, if you have a system. Yeah. that the user provides, then it's focusing more of its intention on the end of the chat. So it's not adding the information in, it's just how do I use the facts. Yep, I got it. OK, I'll repeat that. So the clarification was thinking about using fine tuning more to govern the behavior of the agent rather than to encode factsode facts which is a good it's a very good clarification because I was gonna say using fine-tuning to encode facts I think a lot of kind of literature as points that being a bad idea for a lot of reasons it's costly you have to continually fine-tune as facts change so let's dispatch that I think that's a kind of a not a great idea but you make a very interesting point about fine tuning to govern behavior. Now there's a paper called Raft that came out kind of recently. And actually, as far as my understanding is, I haven't played with it in life, it's fine tuning to kind of do what our notebooks show today of this kind of like, look at the documents that are retrieved, reason if they're relevant, and then don't use them if they're not automatically filtered. It's doing exactly what we're doing in this land graph thing, but it's kind of achieving that same outcome through process fine tuning. So that's a very good insight, you're right. It seems promising to have these kind of fine tuned rag agents so to speak, or it wouldn't be an agent, it'd be an LLM fine tuned for rag that incorporates this kind of logical reasoning or what you're saying like maybe some kind of reasoning about if you have a multi-term conversation like avoid recency bias whatever it is that seems like a very good trend and interesting the challenge is a little bit if you're fine-tuning yourself fine-tuning is hard and somewhat advanced and all that. Alternatively, if it's like a very niche use case, like the RAF system would basically, if fine-tunes to kind of do this, as models get changed or update all the time, you kind of need to keep your fine-tune model up to date if you see what I'm saying so I think I'm still a little bit queasy about using fine-tuning even in that context because of the challenge of keeping it kind of up to date with a state of the art but it's interesting I think the raft paper is a good reference in this direction and does exactly what we do in this workshop but I believe it fine-tunes this into the model or attempts to which is a very intuitive thing to think about basically let the model reflect automatically on retrieved documents and automatically filter them for you. Seems like it should be able to do that. Seems like a good idea. But my hesitation would still be, well, if I want to switch my models, I need to re-fine tune. I want to use Lama 3. I have to fine tune Lama 3 on this task. I can't use proprietary. Maybe I can fine tune. GBD-40 might have fine tuning now. I'm not even sure. I could fine tune Lama 3 on this task. I can't use proprietary, maybe I can fine tune, you know, GBD4 might have fine tuning now, I'm not even sure. I can fine tune GBD, you know, so again, if I fine tune myself, that's hard. It still feels a little bit like I'd rather just sit up a simple, like, orchestrated agent that does it rather than rely on fine tuning, is my sense. Yeah. I guess I'm not a really bad person doing what Google meant to brag to do that fine tuning. Yeah, the provider. Right. Right. Right. Yep. Right. Right. Right. Yeah, okay, so that's a very good point. I think this is also a very big debate. So OpenAI just did an acquisition this week under a retrieval company. I forget the name, Rockset, I believe. So I think they are moving more in the direction of retrieval. I could absolutely see them offering an API that potentially does retrieval for you and incorporates some of these ideas for you. So how much does it does get pushed behind APIs and they take care of whatever is necessary behind the scenes for you? That much does this does get pushed behind apis and they take care of whatever is necessary behind the scenes for you that could absolutely be the case i would not be surprised if at all they move in that direction and i think there's always you know it's an interesting trade off like how much are you willing to kind of um you know abstract behind an api versus not i think there's always a lot of companies developers that want to kind of control everything themselves and build themself have full transparency, and others that don't. And so, you know, it's an interesting question. But of course, for certain functionalities, multi-modality, very few people are going to stand that up themselves. You kind of let that live behind an API. So what do you allow to live behind an API or not? My only concern is, I think for some of these kind of things, they could be very domain specific Like what you consider relevant or not could be very relevant to you and your application You kind of want to be able to control that's the only thing I can imagine It could be kind of hard to abstract that all behind an API which I think is maybe why open I Hasn't done too much in retrieval yet. It's just a hard beast. I know they've been trying for a while Um, I don't know. It's a great hard beast. I know they've been trying for a while. I don't know. It's a great debate though. Yeah, we can discuss more. Yeah, it's a good topic for sure. Yep. You mentioned about long context windows. Oh yeah. So there's a problem of like, lost in the middle. For example, the precision, like, it's, for example, in the context, they'd say 95% is relevant information, 5% is somewhere in the middle for example the precision like it's uh for example in the context they say 95 percent is relevant information five percent is somewhere in the middle and somehow at that point like you have any questions over that yeah yes the question is on lost in the middle of long context i actually did a whole study on this with greg cameron yeah it's a really interesting topic so the insight was basically that long context LMS tend to have lower recall or like you know factual recall for things in the middle of the context okay so that was one observation at least that's what their paper reported so I actually looked at this with Greg and we did something a little bit even harder. We actually tested for multiple fact retrieval So we tested can you retrieve one three or ten different facts from the context and this was using GPT-4 GPT-4 turbo single turn and on the x-axis you can see the fraction of the needles that it basically can get and Then on the y is the number of needles. So basically one needle Three needles ten needles green versus red is basically just retrieving versus retrieving and reasoning So it's like reason is a little bit harder than than than just retrieving These needles are actually pizza ingredients. So basically the background was, you know, this was 120,000 tokens of Paul Graham essays and three secret pizza ingredients or however many, one, three, or 10, but I injected in that context. And I basically asked the LLM, what's the ingredients need to build the secret pizza? So I'd have to find them in there. And basically, as you ramp up the number of needles, go from one to 10 it gets worse so with 10 it's actually retrieval itself is only like 60% so then I looked at okay well where is it failing and that's what I look at here in this heat map so basically this is telling you like how long the context is so a thousand tokens all up to 12 120 thousand and then here's like the needle placement so one to ten so this red means you couldn't retrieve it and what I found is actually it doesn't get them towards the start of the document so the retrieval gets worse if the needles at the front so it's like this I read a book I asked you a question about the first chapter you forgot because I read that a month ago or something same idea and actually I put this on Twitter and then someone said oh yeah it's probably recency bias and that's a good point that basically the most informative tokens in next token prediction are often the more recent ones so you know basically these elements learn a bias to tend to recent tokens and that's not good for a rag so that is all to say I'm a little wary about long context retrieval I wouldn't quite trust basically high quality rag across a million tokens of context you can see look if it's a thousand tokens no problem if it's 120,000 tokens of context you know it depends a lot on where those facts are if they're towards the start you actually can have much lower recall and so that's a real risk which is kind of why it goes back to this whole thing of Like I don't really buy just stuffing everything in the context that far right side I think there's too many issues with bad recall recency bias like you said and So I think until we have very and by the way I also don't really trust you know when they show those needle in the haystack charts It's like perfect. I don't trust any of that. I did my own study I found there's like a lot of errors and I think it depends a lot on a Couple different things one how many needles so in this case you see with one it's okay with ten It's really bad, right? So how many needles and then I saw an interesting study saying that like the difference in the needles relative to your context makes it easier so like in these studies like Pete's ingredients in Paul Graham is really different but if it's just like related slightly it's actually harder still so that is to say I don't really trust a needle in Haystack studies I don't particularly trust stuffing you know passing a million tokens of context and counting that and counting on that to just work effectively I'd be very wary about that that's kind of my thing but you know in these studies look a thousand tokens of conscious already you know that's that is if you're stuffing a thousand tokens yeah I mean that's actually still pretty small so yeah I just be wary about very retrieval from very large contexts yeah yeah okay that's a great question with agents the number of tools this is a really big issue I hear mentioned a lot so So if you recall, if you go back to the agent stuff, so you're basically binding some set of tools to your LLM, right? And that's what we show here, right? I've seen a lot of issues with a large number of tools. So I don't know exactly what the exact cutoff is but this is one of the big problems with open-ended tool calling agents is if I am basically selecting from 20 different tools I actually I maybe the Berkeley leaderboard has data on this as someone knows feel free to mention it but a reliability of tool calling even with a small number tools like on the order of five is already challenging if you're talking about hundreds of tools or dozens of tools I don't think there's really yeah I think it's quite challenging and which is why I've seen more success in a not using these open-ended style tool calling agents, laying it out more explicitly as a line graph where the tool calls live inside nodes. And you're not relying on your agent to pick from like 20 different tools. So you can lay out more of like a control flow where you route to different tool nodes based upon the logic. So that's kind of one thing I've seen. Another thing I've seen is maybe multi-agent type things we have different agents with subtasks with each having like maybe a small number of tools but basically what I've seen is it seems to be that okay maybe it's two things selection from large number tools is definitely challenging one of the most interesting things I saw is something like you can use something like rag where basically you can take a description of your tools create a natural language description embed it and then use basic like semantic similarity search your query versus the embedded summaries to select using semantics that's actually not a bad idea I would actually use that more than I would trust in LLM just like do the tool selection from a list of a hundred that's not gonna work so actually I think that like rag for tool selection is a cool idea I was gonna do a little like test that out and do a little tutorial so actually maybe I'll just make a note of that that's it that's a great a great question to do rag for many tools right yeah well I think I think using semantic similarity for tool selections is a good idea, definitely. What do you mean by data querying, though? Right. Let me make sure I understand. Right. Let me make sure I understand. I think the way I would think about it is, so you know how in the notebook, like in the code, for all of our tools, right, you have this little tool description, right, like retrieve documents, grade them them run web search I would actually write verbose descriptions for all my tools and index those descriptions right or embed them and then I would do and I would probably create like very very verbose high-quality summaries of what the tool actually does and then do semantic similarity search against those summaries I think that could actually I haven't done that yet but I think that could work really well because it's a very tall task asking LLM to differentiate between like 20 different tools, whereas you can do something somatic similarity that would actually probably be very effective. Right Yeah Yeah. Yeah, yeah. So the question was about multi-agent context when you want to orchestrate a large number of tools, having sub-agents with specializations that manage some small set of tools each, and how do you kind of Move between them. So if you look at our land graph repo, we do have a sub directory. It's under It's under Land graph examples multi-agent. We have a few different notebooks that have multi-agent style kind of layouts Which I would encourage you to look at. I haven't personally done too much work on it It seems promising, but I haven't played with it Multi-agent in general and these all reference papers you can also look at the papers a multi-agent in general for production setting feels quite aggressive Although that said as far as I understand I remember looking at the code a while ago Devin and some of the software agents do use like multi-agent style setups So Maybe have a look at the Devin repo or there's open Devin Have a look at these notebooks those could all be useful if you want to learn more about multi-agent Yep Yeah, sure. Right. How do you think about when it makes sense to wrap the rack inside of an agent versus just making a chain on its own system? That's a classic question, yeah. So the question was, when do I use a chain versus an agent? So that's very good. So we kind of touched on it a little bit kind of here so I think that the intuition behind why where and why an agent can make sense is simply that sometimes you want your application control flow to be variable and if you want some flexibility within your application an agent is a nice idea and so all this self-corrective type stuff we're talking about the corrective rag thing. Those are all kind of agentic flows Where the control flow depends upon the grading of the documents and so You know Historically people have largely been building chains and chains are very reliable and they're easy to ship and all that I think with things like Lang graph and of course, yeah, we're gonna line chains I'll speak my book about Lang graph but I've really used it quite a bit and I found to be reliable and I we are seeing a lot people starting deploy with it because you can actually ship and deploy a reliable agent with Lang graph and so I think a blocker to the ability to kind of ship agents has been reliability I think we're I would actually encourage you play with the notebooks and look at Lang graph because it does allow you to have that kind of reliability whether it be necessary to ship something in production. We do have customers that have land graph in production. Whereas a React agent in production is not recommended. Yeah. Yep. Sure. 100%. Yep, sure 100% So okay, so I think yeah, so the different why would you ever want kind of like an agent be it Langrath React or otherwise versus not and I get again I think it goes back to do you want your application of any kind of adaptability so okay here's one we can talk about routing I have three different vector stores I want to build route between them that is kind of a quote-unquote agentic use case because the control flow depends on the question that's one you might want routing you may want self-correction so that's kind of what we talked about here a whole bunch with a corrective rag stuff so you want routing onecorrection. I mean, those are two obvious ones in the context of rag itself. I mean, that's one thing I've often found problem with the rag systems is the routing thing is a real issue. You want it to be flexible enough to deal with questions that are out of domain for your vector store. And you need some kind of dynamism in your application to handle that so looking at the questions like okay just answer this directly don't don't use a vector store so those are like the most popular ones self-correction or routing yeah yeah building evaluation data sets, like question answer pairs are so domain specific. Yeah. I'm wondering if there are like general best practices, mental models, like things to think about when sitting down to build an evaluation data set. Yeah, yeah, okay. So the question was about kind of building eval data sets. Okay, that's a great question. It's often a very, very challenging part of app development. So if you have a RAG application that's domain specific, then oftentimes you have some set of canonical question answer pairs you care about. You know it's hard to find like very, very general rules for that, I think it depends on your application. I think there's kind of this hurdle, any evaluation is better than no evaluation, so small scale eval sets that you can use and just work already better than not doing any evaluation so I mean for this particular case I just looked at the document now maybe maybe I'll back up and answer so so one thing I've seen and I've done this a little bit is you can use LM assisted QA generation so here's one thing you can do. And I've done this a little bit with Langchain docs. I can build a prompt that says, given this document, produce three high quality question and answer pairs from it. And I can just basically load my documents and pass them into that LLM. I use a high capacity model like Sonnet or 4.0 and have it generate QA pairs for me and then audit them. That's a nice trick. I've used that. It actually kind of works. Now, you have to be careful with it. capacity model like Sonnet or 4.0 and have it generate QA pairs for me and then audit them. That's a nice trick. I've used that. It actually kind of works. Now, you have to be careful with it. You only would pass it usually one document at a time to keep it really restricted. And you audit them. But that's a nice way to bootstrap your Eval sets. That's idea one. And that gets into the whole idea of synthetic data sets. But if you're building domain-specific synthetic QA pair data sets, that's that's a nice trick so basically using Ellen to help bootstrap I think that's one idea that can help a lot yeah and otherwise I think that basically trying to stand up a small evaluation set for example for rag is even this case five questions but you can already see I can get some nice insights it's very simple to set these up. I have my little experiments all over here. And again, it's only five questions, but it gives me some immediate insights about the reliability of React versus versus Landgraf agent. So keep it small, potentially use synthetic data, start with something and then like kind of build out over time. Now, a whole other thing here is we didn't talk about this too much but if you have an app in production then the way this whole thing kind of comes together is you can actually have different types of evaluators that run on your app online we call those online evaluators okay so this is with our internal app and this gets back to the question I think he mentioned of you can have a bunch of evaluators for RAG that don't require a reference so like I don't show it here but but basically I can look at like document retrieval quality I can look at my answer relevance or hallucinations I can run that online I can flag cases where things did not work well and I can actually roll those back into my eval set so if I do that, then I actually have this self-perpetuating loop, like Karpathy talked about, like the data flag wheel, where actually I'm running my app in production, I'm collecting cases of bad behavior that I'm tagging with online evaluation, and I'm rolling those back into my offline eval set. So what you do there is look at the case that the app is doing poorly in production, audit them, correct them, so build a canonical question as a parent from that and put that back in your test set. And that's a good way to bootstrap and build it up. So I'd start cold start, synthetic data, small scale examples, online evaluation, or some system to check online where it's failing, loop those back in and build it up that way. That's like your data flywheel. Yeah. Actually, I even had a slide on this in one of my older talks I used to work in self-driving for many years and I actually was a big fan of Carpathy stuff at Tesla and actually I've had his his thing here of this is like the the data engine thing of like you know you ship your model you do some kind of online evaluation where's it failing capture those failures curate them put them back in your test set run that as a loop that's like he called operation vacation because you can go on vacation and the model keeps getting better and this is more in the context of like training models because basically all those those failed examples once they're labeled they become part of your your training set but the same thing applies here with LLM apps cool yeah Same thing applies here with LLM apps. Cool. Yeah. So I just wanted to ask, you just mentioned fix to SQL. Oh, yeah. So I just had a question where, for example, all that happens when we send stable schema, and we send this SQL query, and then we have the database. And if the result is too large or something, so we cannot send it for another LLM user-friendly query. Like user-friendly after or something, right? How do we handle that? What do you suggest how the query is so big that we cannot put it in the context. Yeah, the question was on text to SQL. We actually have a pretty nice text to SQL agent example here. So it's in Langrath examples. I think it's in SQL. Where is it? I'll find it here. What is it? Tutorials, as in tutorials. So SQL agent here. I think a lot's in the prompting. So basically, in this particular case, I believe Ankush from our team set this up. You prompt your, you can do a couple things. So you can prompt your SQL agent to, where is it? It's somewhere where he tells it to, yeah, it's based on all these instructions. Be very careful about, let me just find the, in short, you basically can instruct the SQL agent when it's writing its query to ensure not to extract an excessive amount of context. And I can't remember exactly where it does that. Okay, yeah, it's right here. Limit, always use a limit statement on your query to restrict it to like whatever it is, five results, and that's the hard-coded thing here. And then also in your SQL agent, you can incorporate a query check node to actually look at the query before you execute it to sanity check for things like this. So basically, I would have a look at the LangGraph example tutorial SQL agent notebook to have... I've actually... I ran this and evaluated and I found it did work pretty well. So that's one thing that I would look at. One way question on SQL, a lot of times like the SQL, they don't follow the group, I have questions, but how do you handle filtering questions, like where flaws, like when you have a high cardinal column, like I have a question, right, like if you say, I'm going to SQL query and it needs to be where, where flaws in SQL, where exactly translates to like any column value, like any value from the column, but like especially if in some depth, but you're high cardinality, like maybe 500 values, like how do you handle that? That's what I think. Hmm, yeah. So the question was related to like, how do you handle high cardinality columns? And I guess that is related to restricting the output size. I mean, I'm actually not really a sequel expert so I'm probably not the right person to ask about very gory details of text to sequel but in general I would kind of consider can you prompt the LLM effectively like okay two different things one the general ideas here were basically upfront prompting of your LLM to kind of follow some general kind of query formulation criteria. And then two, an actual query check explicitly to review and confirm it doesn't have some of the issues like extracting accessible context. But for anything more detailed than that, I'm probably not the person for those insights. But there might be a specific text to SQL deep dive at some other point in this conference and you should definitely seek that out. But I haven't done that much of text to SQL. Yeah. question answer rag patterns, but one thing I'm interested in is more long form document generation, so writing for a floor or something like that. Yeah. Can you see the interesting patterns of what you think about tools marketing solutions with a line graph or other types of things to achieve that type of outcome? Yeah, okay, so the question is related to kind of document generation. That is a really good theme. So we actually have kind of, there's an interesting paper, I did this a while ago, I need to find it. Where is it? So we have a notebook, if you look in Landgraf examples, Storm. So this was actually for wiki article generation. Here's kind of the diagram for it. We actually have a video on this too. I'm actually trying to refresh myself. I did this like three or four months ago. But it was basically a multi-agent style setup in line graph. Where if I recall correctly, I'm just looking at the flow here myself. But basically what it did was you give it a topic and it'll kind of do initially this kind of like generation of related topics and actually uses the multi-agent thing of like editors and per and um experts the experts go and do like web research don't worry too much of the details so the point is it was an interesting paper in flow for wiki article generation using land graph in a multi-step process and actually the wikis are like pretty good i think at the bottom of the notebook i have an example wiki article generation using line graph in a multi-step process. And actually, the wikis are pretty good. I think at the bottom of the notebook, I have an example wiki. So you can see all the code here. That's the graph. Yeah, and then here's the final wiki that you get out from this type thing. So it's pretty good. Have a look at that notebook. I think, yeah, Jason Liu also had a post on this recently, this idea of like report generations, a theme that we're going to see more and more of. This was one idea that was pretty sophisticated, though. You can probably simplify it a lot. I've done a lot of work just on like a simple kind of distillation prompt, like perform rag and then have some generation prompt to give a bunch of instructions for how I want the output to be formatted that could also is really effective yeah Yeah, so the idea the question was related to user feedback. That's a really good one. So we do have, I think I mentioned previously, if you look at LandGraph, where is it? We, I believe we have some user feedback examples. Thursday, we're definitely going to be announcing something that has a lot of support for user feedback and I would encourage you to keep an eye out for that. So Tarasyn is going to launch that here on Thursday. I will look for, I know we have some user feedback examples in LangGraph, but I will need to find them. Let me see. We probably, let's check Lang, We probably tweeted about it at some point. I haven't actually done anything with user feedback, though. Let's see. Yeah, maybe I might have to get back to you on that. I thought we had some nice examples with LangGraph. Yeah, I'll have to get back to you on that one. Feedback. Customer support might be in here. Let's try something else. the LandGraph docs. We have a bunch of tutorials in the docs here. Just Google LandGraph documentation. I'm just poking around here for user feedback. Ah, here we go. Look at this. So LandGraph how-tos, human in the loop. I would have. Look at this. So, LandGraph, how to's, human in the loop. I would have a look at that. I have not played with that myself,, yeah. Okay, right. So for like mid to longer term problem solving tasks, how do you incorporate user feedback to ask for more information? So I would have a look at this documentation because I would imagine it will cover examples along those lines. I haven't personally done that. I also believe that I would have a look at the customer support bot because that's an example of a kind of multi-turn interaction between a user and a support agent so I would look at the customer support bot which will on my team did as well as the documentation on human loop so those are two things I would check out there nice yep to model architecture for training models for agentic reasoning? For agentic reasoning? Yeah. Yeah, so that's kind of an interesting question. So the question is related to training models specifically for agentic reasoning. If anyone, so I mean there's a lot of, there's a lot of work on prompting approaches for different types of reasoning for sure. I'm a little bit less familiar with like efforts to fine tune a model specifically for particular like agentic architecture or use case, but you could imagine it. Most of the work that I've encountered, though, is just using high-capacity generalist models with tool calling and specific prompting techniques. So React is kind of a particular orchestration flow and prompting technique rather than, you know, you can interchange the LLM accordingly. I think the main thing typically or historically for agents has been the ability to perform high quality and accurate tool calling. Because agents, that's one of the central components of agents. And so that's kind of been the gating thing. And I think model providers have been focused a lot on just high quality tool calling, which helps kind of like all agent architectures. I haven't seen as much on fine tuning for one particular architecture. I think it's like high capacity journalist models with tool calling and then prompting. So it's more like in context learning. That's kind of the trend I've seen at least. Yeah. Can you talk a little bit more about line graph checkpoints? Yeah. Yeah. Yeah. So the checkpointing stuff,? Can you just hold it? Yeah. So the checkpointing stuff, actually, this Thursday, that's going to be a lot more relevant because we're launching some stuff to support deployments for line graph, in which case you can do a bunch of different things, but you can have a single state that persists across many different sessions. You can also have checkpoints. You can return to state and revisit an agent from a particular point don't worry about that too much for now I think there'll be a lot more documentation and and kind of context for that on Thursday when this stuff through deployment comes in but it's good to be somewhat aware of I'll poke around the documentation for a little bit more on checkpointing, but it really becomes relevant on the stuff we're announcing on Thursday. So I would have a look then. Let's see if we update our docs. Yeah. Yeah, so there is some documentation on it now, but it'll become a lot more interesting and relevant come Thursday and we have a lot more support for deployment Yep Yeah Yeah or makes small mistakes in YouTube, for instance, instead of YouTube video around it, it says YouTube video from the web. Yeah. To stop the process,. OK, yeah, so that's a really good question. And so the way it works with the existing, so it depends on the architecture. So using the React architecture, let's see if I can find an example of it so here's with react agent let's look at one of the traces see if I have an example so basically the tool call itself will return like an error and the LLM then is expected to self-correct from that error it has to kind of self-correct from that error. It has to kind of self-correct. So that's kind of one approach that at least we do with the React agent. So actually, you can see it in the notebook. If you go to, and if I can find some traces that have that example, I will pull them up but I think it's in utilities somewhere yeah so basically this tool node with fallbacks basically what happens is in this tool error so this if there's an error in the tool call itself it'll return that error. And usually the agent will then, or the LLM assistant, will look at that and self-correct its tool call. So that's typically how it's done. And this actually is reasonably effective. But again, the nice thing about the other implementation, the custom agent I called in the notebook, is you don't rely on tool calling in this way and so you can get around this type of issue but basically catching the errors in the tool call itself with this code is what's currently done let's see if I can actually find an example yeah I mean if I can look for one where it gets the answer wrong. Yeah, let's see this one. Let's see if we can find a tool call failure. So here's the trace. Let's see. Okay, this didn't have a tool call error. Yeah, so basically what you'll see in the message history is that like the tool itself will return this error message and then the LLM will say, okay I need to retry and then it'll retry and hopefully get it right. Yeah, yep. Oh yeah. Yep. Yes. which is great for that exact problem. Yep. It does like the instant revalidation of the output, especially for those simple little errors. Yes. I'm wondering, well, first of all, it'll probably work, but does Nightchain, I haven't used it much, so could you use it with an instructor and then I can't take a look at it? And it's similar to the light. Okay, so this is a very good point. So yeah, I'm a big fan of instructor. I haven't used it as much, but what you're saying is one particular type of tool call. So basically that pertains, I believe, more to structured outputs, which is indeed a kind of tool call. And when you're using something like a pedantic schema, you're right. It's very easy to check and like correct errors. So I've found catching errors like with schema validation, like using a structure is really good. And we have some other things you can use within the line chain to do the same thing. So that's one type of error that's actually particularly easy to kind of detect and correct. What we show in this notebook here, and the code I showed is more for any general tool. So this code here will operate on any tool you call regardless. So it doesn't have to do with structured outputs or anything. And so just a more general check for tool call errors. Now in terms of instructor with Langchain, now maybe I'll just back up a little bit. So LangGraph does not require Langchain at all. So that's kind of point one and neither does lang smith. So actually everything we're doing here does not need to use lang chain. So actually that could be a pretty interesting thing to try for like kind of the choose your own adventure thing. But basically in the custom agent part, I use with structured outputs to do the grading so if you go to yeah if you look at the the retrieval grader here so this is using LLM with structured output and here's my grade schema try that one instructor that should work work great. You don't need LangChain at all for this. And that'll fit right into the LangGraph. So actually I think it'd be great to use instructor with LangGraph for this particular use case. And I do agree that Lang, that instructor is really nice for those kind of like schema validation error correction. Plug and play. That, that I'm gonna make a note of that's a really good kind of choose-your-own-adventure case or should I put that try instructor with line graph for grading yeah I like that a lot yep yep Yep. Yep. where is the not, where is the rag pipeline to be in the world? Like what are we still not doing at? And your guys are thinking, or so, down in the moment. Yeah. So a question was related to just rag in general. And where is rag? Like rag agents. Rag agents. Yeah, sure. Well, to be honest, a lot of the problems with RAG, I think about our own internal application chat Langchain, a lot of problems with RAG actually are retrieval problems. Retrieval is just hard. I'll give a good example. Like Langchain, we have, I'm trying to remember, five million tokens of context across all our docs, something like that. We have also we have a very long tail of integration docs. You want very high quality coverage and questions across all of that. There's a lot in how you index all that stuff to ensure that you boost retrievals from more canonical how-to guys that are much better documented, but still having coverage over long tail content for like, you know, long tail questions. For example, if you're using raw semantic similarity search, you could have relevance to, you know, say your how to guide, which is really well developed and three random long tail documents that are not well developed and they'll all get returned. And so how do you overlay different systems? It could be re-ranking to basically promote content that you believe to be more accurate or better based on some criteria. So that is all to say, I think with RAG, the challenge is actually just domain specific retrieval for your application. That's just a hard problem. And there's been a lot of work on this. It's been around for a long time. I think that's really the limiter. And actually there's kind of no silver bullet. Like in our case, we're having to look at the structure of our documents very carefully, design our retrieval strategy based on that doc structure. Like in particular, we're thinking about applying certain post-retrieval ranking to docs of certain types based upon their importance or thinking about retrieving a large, like a large initial number of docs and then boiling them down with, with kind of re-ranking based upon importance. So I still think retrieval is very hard. It's very domain specific. It depends on the structure of your documentation. There's kind of no free lunch. I think the things that are good about RAG is context window is getting much larger for LLMs. And so back to that point I was making before, I think we're seeing, and we're considering this ourselves, less worry about the exact right chunk size. You can think more about chunking in different ways and then passing full documents to your final model. So I think that part of it's really good. But still, even like, even in this particular case, you probably still need some re-ranking to promote the most important documents. So I think retrieval is still quite hard. In particular, like looking at the line chain docs in particular, the overlay of document importance on top of raw semantic similarity search. Take a case of like, I have a question, semantically, it's similar to 10 different documents. Those documents though vary widely in their quality and they're relevant, like more like higher level relevance. Like maybe that passage is related, but like it might be a general question, but how do you build an agent? And then some random integration doc talks about building agent for integration X. And I wanna make sure that the more canonical, well-developed agent, you know, overview doc gets promoted and passed back in the end. Stuff like that. Sorry, it's a long answer, but basically RAG is, it's hard. I mean, I think retrieval is really the hard part. The generation part is getting better and better as long contexts grow. Yep. Lance, so this pre-ranking approach for your documents, what's the metadata? Do you have a relevancy to a particular topic as well as numerical ranking? Yeah, okay, that's a great question. So the question was, when we talk about this re-ranking, how do you assign this relevance to your documents of what is that? So I'll just give you what we've been thinking about. I actually think it is for us going to be a hand-tuned kind of relevant score based upon our doc structure. So if you look at the Langchain docs, like go to Langchain documents. Yeah, so language documentation. We have these sections up here, tutorials, how-to guides, conceptual guides, which are like really well developed, more recent, well curated. These you can imagine have some kind of relevance or importance ranking of one or highest. So these are documents that contain very high quality, well curated answers that we want to promote and serve to users in the generation phase. However, let's say someone asks a question about one particular integration, right? If you go to integrations, we have all these pages, right? Components, go to retrievers, look at the ZEP cloud retriever. This is some stuff related to ZEP cloud specifically. If someone asks about ZEP cloud, you do want to be able to retrieve that doc, right? And so some ability to differentiate between questions that need, you know, general answers, in which case you would promote your more canonical how-to guides conceptual docs versus questions that require retrieval from very specific integration docs, in which case you would still promote this information. That's kind of the crux of it. And I think we'll probably use kind of manual or heuristic scoring to up weight or up rank our core like how to guides and conceptual guides over longer tail integration docs. And we might have a router that will indicate whether the question is general or specific. So those are the two things I'd probably do. So routing on the question side and then some kind of heuristic relevance or importance grading or quality grading on the document side. And that can be packed in the metadata that you pack along with your index chunks. Yeah. I don't know. Maybe not. Yeah. Oh, yeah. So let's say a typical RAC application where there's a question and answer but we kind of maintain the multi-turn, like for example, we maintain the compensation history of the user to kind of create a. The problem is, like, let's say, for example, a question is asked and then the retrieve chunks are, like, let's say, for example, a question is asked, and then the retrieve chunks are, let's say, five. Right. And then a subsequent question is asked, so let's say, no, but it's related to the first question, but still somehow in the first node, you transform the query, and then the retrieve chunks are still the same. So I would just get that answer, which is more of the first answer. So how do you, my question is, how do you make sure that, let's say, he wanted to deep dive into the document, like into the more context, how do you make that happen? Yeah, so the question I guess was like in a multi-turn RAG context, let's say the case where a user asks an initial question and you retrieve some documents, you produce an answer and they ask a follow-up that says give me more information about this. Now, do you want to re-retrieve or do you want to re-reference those same docs? No. So what happened in my case is I try to rewrite that question. OK, you do a rewriting. So you rewrite the question. OK. And then go to the documents. So most often, the documents would be the same as the first. The same as before. Yeah. OK. So the answers would be mostly the same. OK. OK, interesting. So the problem there is more of a retrieval problem. Okay. The answers would be like, what's the same? Okay. Okay, interesting. So the problem there is more of a retrieval problem. You're doing a rewrite. You're still retrieving the same set of documents, though. Now, what do you want to have happen? Do you actually want to retrieve different documents, or do you want to? It's kind of like deep dive. Yeah, but that's a question. What do you mean by deep dive? Like you're retrieving, let's say it's a chapter of a book, you're retrieving only the first page, you want to retrieve the whole chapter. Yeah. Okay. Okay, then I think actually a question rewrite would probably not sufficient. What I would think about more is for that second pass You could actually do something like metadata filtering on your chunks if you have your data or your documents partitioned by like Chapters or some sections. I would just do a bulk retrieval the whole section or something like that So it's more like a trick on the on the retrieval side rather than a rewrite of the query Because I hear I see what you're saying you rewrite the query You might get the same Docs back if you want to guarantee that you actually get like a deeper dive in your Docs Then maybe it's something in your retriever itself. You can increase K. So retrieve more Docs You could use metadata filtering to like ensure you get all the Docs in a given chapter So I think it's more retrieval thing. But that's kind of an interesting point, though, yeah. Cool. Well, I know it's been two and a half hours almost. So there we go. It's good. Yeah, yeah. Yeah, I'm hanging out for another till noon. So cool. Yeah, yeah. Okay, okay. This is good. So, see, the question was he's doing the local agent tutorial, and the question's on the eval set. So, actually, that's a fun one. Modify them any way you want. The key point was I wanted some questions that are definitely outside the vector store. So I asked something about, like, two things about sports, because I know it's not in my vector store about agents. So I think I indexed three blog posts about, about like agents and prompting and adversarial examples I just wanted some orthogonal questions that'll force web search so that's the only thing there but you actually play with those and you can modify them and all that yeah but that's cool. It's working. Are you using Lama 3? Yeah. Cool. Yeah, I was doing the 70B. Oh, you have a laptop big enough for 70B. Not really. Okay, okay. You're at the edge of... There's an M1 that's taking like a minute. Okay, okay. And then I switched to the 8B. Yeah, 8Bs. Yeah, exactly. I mean, it's actually kind of nice. You can even run the 70B, to be honest. I'm not sure I can even run it. But, yeah, that's actually kind of nice. You can even run the 70B, to be honest. I'm not sure I can even run it. But yeah, that's cool. Nice. Let's see. Well, I can just hang out and oh, yeah. Sure. What would be the best way to incorporate full-on questions? Is it pro to your... Yeah. Well, if you have a chat application, so the question was related to how do you incorporate multi-turn. So if you look at the React agent agent it uses a chat history as its state in that case follow-up questions we captured just in the message history as part of chat the I think the current layout of the custom line graph agent though is a little bit more single single turn so it'd be kind of question and answer. That's, oh yeah. I meant that the agent actually has to come back with a question to talk about details. OK, got it. So the question was, how do you modify the agent so that it will actually return, like, if it needs more clarification from the user? Yeah, these particular agent examples don't do that. But again, I think that's maybe a good takeaway for me. I should add that to these tutorials, incorporate a simple example of multi-turn. I will do that and get my contact in. I will send that to you. So now I have a simple ev eval or pull off questions. Yes. I think that should be the smarter thing. Yes, exactly. So you want, so I mentioned previously, if you look at Langrath, let me find it. It's one of our notebooks. The customer support agent. So Landgraf examples customer support is an example of an agent that has like multi-turn dialogue, but it's complicated. So I'd like to maybe augment these tutorials with a simpler example. Yeah, I'll follow up on that. Give me contact info, and I'll send you something. Cool. Well, I'll just sit up here. Anyone can just come and grab me. Thanks for everything. Hopefully, the cookbooks are working. Yeah, it was good. Made it two and a half hours. So. Thanks. Субтитры подогнал «Симон»!
Architecting and Testing Controllable Agents: Lance Martin
LLM-powered autonomous agents combine (1) Tool calling, (2) Memory, and (3) Planning to autonomously perform tasks. While they hold tremendous promise, agent reliability has been a barrier for large-scale deployment and productionisation. We’ll cover ways to design and build reliable agents using LangGraph, which can support diverse self-corrective applications such as RAG and code generation. But, just as critically, we’ll cover ways to use LangSmith to test your agents, examining both agent's final response as well as agent tool use trajectories. Collectively, we’ll talk about three types of testing loops you can incorporate into your agent design process - at run time, pre-production, and for production monitoring.

Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025

About Lance Martin
Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025