 Hi this is Lance from Langchain. This is the 11th video in our Langsmith evaluation series focused on summary evaluators. So the motivation for this is let's say for example I have an evaluation for document grading. Now we talked about this previously I've used this quite a bit in the context of RAG where I basically do retrieval and I grade each document to determine whether or not it's relevant to the question. So I basically have an LLM grader that does this and it returns a binary score of yes or no. Now I've built an evaluation set for this previously and we've actually looked at it a little bit. It's called relevance grade. If you look at the examples, each example has a document, an example question, and a score as to whether or not the document is relevant to the question. This is just a nice little toy data set we can use to test this grading process. So there's three examples. You can see one is no, because the question and the document are not related. So it's an example of an irrelevant retrieval. So here's kind of the question though. How can I create a metric to summarize performance in this data set in a custom way? So you kind of saw before when we run our evaluations, I can go back to the data set, I can look at the experiments, you do get this correctness score. Now all this is is just simply a summation of the scores from each individual experiment or actually I guess it's a mean in fact. So it's basically the mean of scores across your experiments right in this particular case. Now what if I want a different thing? What if I don't want for example correctness to be my aggregate summary metric across this whole data set? What if I wanted to find something custom? That's the motivation here. So let's kind of just like kind of see where we are in landscape. So we've built that the data set we just showed. So it's a developer curated data set or manually curated data set. We're using LM as a judge evaluator. I have ground truth and a performing evaluation with LM as a judge. Now, the only thing different is that I'm going to define a custom metric for my evaluation across the data set. So this is the overall flow. It matches what we've talked about this before already. But the only trick is I'm going to define a custom metric in this particular case. So here's kind of the challenge I'm going to pose. First, I'm going to find a grader using OpenAI. And actually, we've seen this before. I define a data model that returns a pydantic object that's basically a binary score of yes, no. I bind that object to my LLM right here. So it's going to produce a structured output. I have my grader prompt here. And that's it. That's my grader prompt here and that's it. That's my grader chain. And then I just have this function predict open AI, which will take a dataset input and it will just invoke my grader and return my grade as an output grade, which will just be yes or no. That's all that's happening here. Now what I'm gonna do is I'm also gonna find a misdraw grader. Same flow, but in this case I'm gonna use misdraw'm also gonna find a misdraw grader same flow but in this case I'm gonna use misdraw running locally in my laptop as the grader and I'm gonna use a JSON mode to enforce JSON output but that's actually what we want to test here like can I reliably get a binary score out and here's my grader prompt and so forth so the setup here is that I'm going to do a comparative a B test between my misdraw grad my open a greater on my evaluation set so we've seen that all before nothing new there the only thing I'm going to add is if you go down I want a summary metric on this data set and what I want to do is I want a metric that basically combines precision and recall. So, you know, if you recall, precision basically is the measure of true positives over all your positives. So basically that's true positives and your false positives. OK, now recall is kind of the other side of the coin. Recall is the true positives over all possible positives in the data set. So the intuition here typically is that these are kind of intention, right? If you have a data set and you basically guess everything as a positive, right, just promiscuously, you just guess positive every time, you can have perfect recall, or you by definition will have perfect recall. But your precision will be quite bad because you'll probably have lots of false positives depending on the structure of your data set. And the inverse is also true. So there's a nice metric called F1 score, which basically unites precision and recall. It's basically the harmonic mean of precision and recall. You can look that up. But the intuition is that it combines these two metrics. So it's commonly used in machine learning and other fields as a way to kind of combine the tradeoffs between precision and recall. So I'm going to find an F1 score evaluator. Recall that my data set, every experiment or every run or sample in my data set is yes or no. So I can just go through my samples, and each one is scored as basically yes or no. So here's where I'm doing that comparison. And I'm just bidding them into true positives or false positives false negatives based upon the the route the relationship between my example which is the ground truth reference and my grade prediction which you recall we wrote out grade here in both cases so this the output of our predict function just returns a grade and that's our yes, no score. We compare it to ground truth here. We log the number of true positives, false positives, false negatives, and we compute F1 score using that. That's all we did. So again, we have this function compute F1 score here. And now all I need to do is again, call our good old evaluate method with our particular chain of interest. So this time I'll evaluate misdraw. And I'm going to use my F1 score evaluator. That's it. Let's kick that off. So that should be running now. And that's running against our data set. So you can see we've set our data set here, relevance grade. And it looks like that run. Great. Now we can do the same for openai so that's running now we can go back to our data set relevance grade let me actually do it go here so again i'm in langsmith here this is my data set i'm in the day set relevance grade i can see my experiments and these two tests just rolled in so this is test score openai test score mistral you can see this is the prefix we added to our experiment so that's where that's that's coming from and what you'll notice is we have this f1 score now logged with each experiment which is pretty nice and we can go ahead and open those up so let's open up in comparison view. So this is pretty cool. We can basically look at the summary metric here. So that's cool. We can see that both have a summary metric of one. So they both got all of our test samples correct. And we can confirm that here. So here's the reference. Here's what each chain produced. That's it. Really simple, very nice, very convenient to be able to find custom evaluators on datasets. And you can see right here we're just viewing the both in comparison mode. And we can confirm that the summary metric is indeed correct because it looks like both of our chains got every answer correct. So that's it. Very convenient to use and I recommend exploring with a little bit more. Thanks.
Summary Evaluators | LangSmith Evaluations - Part 11
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video explains how to define evaluators that compute customized summary metrics over a dataset.

Documentation:
https://docs.smith.langchain.com/evaluation/faq/custom-evaluators#summary-evaluators