 hi this is Lance from LagChain this is our fifth video on lagsmith evaluations so our first video kind of laid out why evals are important and interesting our second video laid out kind of the core langs with primitives that we'll be working with we just talked through two two important concepts so building a data set from like a set of manually curated in our case question answer pairs um we built a data set based on this blog post about the new databricks model and i basically manually built a number of question answer pairs from that blog post i add them to my own data set that data set then was called dbrx and i used the sdk to create it so that's really it i also showed how to build a data set for user logs which is really useful for you if you want to take a bunch of actual user data like user questions um and convert them into like a data set with ground truth responses for future testing so that's another you know really useful and common technique for data set building so now let's get into uh evaluation so here's a question i build my data set how to actually evaluate my lm against it so in the second video we talked about this information flow but i'll just reiterate it briefly so we have a data set the data set has examples in my case the data set has input output pairs question answer what i'm going to do is i may have an app and we'll see that shortly i'd like a little example app um that app sees an input from my data set produces an output i also have the ground truth output in the data set i pass the user or the the ground truth output and the app output to an evaluator, and it'll perform some evaluation and return a score. That's it. Now, here's where it gets a bit interesting. The world of evaluators is actually pretty broad, and we've actually touched on this in a few other videos. So there's custom evaluators, there's built-in evaluators. Within built-in evaluators, there's evaluators for labels or non-labels. In my particular case particular case i have labels and i want to use a built-in language with evaluator so we have a bunch of them listed here and i also go over and show you so off the shelf line blank chain evaluators is is like a really uh nice way to go um so you don't have to kind of re-implement something from scratch for question answering again my data set is question answer pairs. So on an evaluator that operates on question answer pairs, right? Here's a few different ones that are popular QA, context QA, COT QA. The high level point is this, COT QA is often a very nice evaluator because it'll use chain of thought reasoning um to basically look at the llm generate answer versus the ground truth and to evaluate whether or not uh they they match um and so typically in terms for the greater llm you use like a very powerful model like maybe quad opus or you might use you know open aiT-4, for example. But that's the high-level idea. You're using chain of thought reasoning to determine the final verdict. So let's actually just walk through what's going on here. I'm going to pick that COTQA as my evaluator. Remember I built my data set, DBRX? Let's actually go over and have a quick look at that. So if I go over to my lagsmith um i'm going to go data system testing dbrx search for it here it is i have my data set i've done no evaluations against it it has four examples so this is kind of where i am currently now that's my data set name remember i built this function answer data bridge questions so i define that up here just using open ai very simple um i'm plumbing in my data set name i'm plumbing my evaluator i'm adding a little prefix like this is test qa open ai and i'm also adding some metadata metadata like i'm you know website context into you know gp35 turbo so anyway that's all that's going on here and this evaluate is all i need so i kick this off and this will kick off an evaluation so again think about what's happening look at the flow here all that's happening is i have those four questions each of those questions is going in going to my my basic llm which is this answer chain right so that's this thing so each question is getting plumbed into this here Here's a good example, right? Right here. We plumb in a question, we get an answer out. It's just doing that behind the scenes. So we take those four questions, we plumb them in, we get answers out. For every one of those answers out, I go to that data set, I fetch the ground truth answer. Again, we can see them here, right? Look, here's one of our inputs. So this input gets plumbed into our LLM. And we have this ground truth output. That's it. So let's go back. Hopefully that ran. It ran. Now here's where I do. I go to my data set. I look at tests. Now I'm going to move my little head here. So now I have a test. You see this prefix that we added? It is now, right this this thing right here we can see you know our name has that prefix in it we see some metrics here latency p99 p50 p99 um and we can see things like error rate we can see our metric and so forth so let's actually just dive in here so this is where you can really have a lot of fun and do a lot of kind of inspection of your results. So here's what's going on. The input here is the question that I plumbed in, right? Go back to our flow. The input is this thing. It's just my question. All question all right the reference output go back to the flow is this thing it just makes the correct answer okay so i have the input question i have the reference output now here's what my model returned so this is what we're assessing we're assessing this reference versus what i what i returned using that cotTQA evaluator. So behind the scenes, let's actually dig into that. So there's two things I can click on here. This open run thing opens up just my chain. Okay. So this is my chain, again, which we defined up here. So it's this answer question with open AI thing. So that's just this running on our input there's all the context and here's the question that got plumbed in here's the answer so if i kind of go back um that's what that run is that's all that's happening there now i might want to know well what did this great how does greater work what actually happened there so if i click on this little arrow it'll take me to the evaluator run and that's going there now i might want to know what did this great how does greater work what actually happened there so if i click on this little arrow it'll take me to the evaluator run and that's going to open up right here so this is the evaluator that we used off the shelf we didn't have to write this or anything we can actually go we can see we're using open ai as the evaluator which is fine and here's actually the prompt this is very useful you're teaching a quiz blah blah it gives you a bunch of criteria um okay so basically uh what's happening is this is the greater prompt and you're seeing the question and the context and the student answer so the context gives you the ground truth answer the student answer is what the model returned and then here's the output here is like the reasoning and here's the score this is really nice you can audit the grader really easily so if I go back let's zoom all the way back out what's going going on here? I define the data set. My inputs are here. My reference outputs are here. My LN generations are here. My scores are all listed, one or zero in this case. And I can dig into each one and understand what the evaluator did. I can also dig into my generations using this open run to see how they work. So if I zoom all the way back out the stack, what are we doing here? We're doing evaluation against our data set with a built-in line set evaluator. This was the information flow. And if I go all the way down, what do we just do? We had our data set of data bridge examples, questions, four of them. We used LM as a judge using a built-in line chain evaluator against those ground truth answers that we provide in our data set and we basically did an LLM evaluation that's it so we're going to be kind of building on this in the next video thanks
Pre-Built Evaluators | LangSmith Evaluations - Part 5
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video introduces how to use LangSmith's many pre-built evaluators for tasks such as RAG (question answering), evaluating generations based upon supplied criteria, etc.

Documentation:
https://docs.smith.langchain.com/evaluation/faq/evaluator-implementations

Code:
https://github.com/langchain-ai/langsmith-cookbook/blob/main/introduction/langsmith_introduction.ipynb