 Hi, this is Lance from Langchain. This is the ninth video in our Langsmith evaluation series. So the prior video talked about setting up, I'll show over here, a set of test cases in a data set. So here's the data set we set up called relevance grade and it had three test cases. I can show you each test case basically has an input, text an example question and as output like a ground truth assessment of is this question relevant to the text so this is a rag challenge that I you know I work on quite a bit where you want to do document grading in line with your rag flow and what I showed previously is testing various LLMs on this challenge directly from the prompt hub. So you kind of saw over here, we have the prompt playground here. This is my grader prompt. Again, it takes the inputs that we just saw in our data set. And I can easily pull up the data set here in the prompt hub. I can just run with various LLMs you can see over here against this and those are all logged to to their to our data set and we can go inspect those results so that is kind of the workflow we talked through now we talked previously a lot about different kinds of evaluators that we can use to automate evaluations like LLM based graders. And it can be really useful to run these experiments, but actually have an LLM based grader automatically perform grading, right? Because if you remember, we're in the prompt up here, we kicked off an experiment against the data set we selected here, but then we have to go and manually review those results ourselves. What if for every experiment I run here, automatically I have some LLM grader that will just run on those results that I can inspect? So that's possible. And if I go over to my data set here, so here's the data set, here's my examples. If you look up here at evaluators here, you actually can set here. So here's the data set. Here's my examples. If you look up here at evaluators here, you actually can set these. So you see this add evaluator. This is where I can basically define my evaluator. I can set a evaluator model. And the key point is I basically can customize some evaluation prompt, right? And this can be whatever is relevant for the task I care about. And when you basically set this evaluator, it'll always run when an experiment is initiated against this data set. So I've actually already done this. So I have this JSON evaluator, and we can kind of have a look here. You can edit it, which is actually really nice. So this is using OpenAI. I'm using 3.5 because it's nice and fast and so this is what i'm going to use to grade the responses uh that we basically generated here in the prompt playground by testing different lms right so for each lm we kick off an experiment. And this grader then will assess. You can kind of see here the results are logged or written all here. And they're, of course, logged to our data set. But we can actually grade them using an LLM-based grader automatically, which is quite nice. And now here's the prompt. You're a grader. You're going to be shown a student submission for a JSON string and a reference value that should be present in that string. You see a looking score. So anyway, you're laying out the logic of what you want the thing to assess. Really what's happening here is I just want it to assess, is it valid JSON? Is there no preamble? Does it contain the right value for score? That's really all that's going on, right? So I can set this evaluator. It's now attached to my data set. That's really all that's going on, right? So I can set this evaluator, it's now attached to my dataset, and when I ran those experiments in the hub, now if I go to the experiments, you'll see this correctness column. So this has automatically been populated for you every time you run an experiment from the hub. And if I zoom in, this correctness captures what that automatic evaluator assessed about the output. So I don't have to manually review them per se. In this case, the LLM-based grader, which I defined and pinned to my data set, automatically runs a new example. And if you recall, we've shown this previously, but you can actually click on these and open up the grader traces. You can actually look and see exactly what the grader saw. So this is a prompt that the grader saw. You're a grader. This is what we saw before. Now here's what's interesting. It actually inputs the reference from the data set and the submission from our experiment. So the submission is what the LLM produced. The reference is like the correct value and again this just shows like the logic you know ensure that's proper JSON and so forth and then we can actually look at the scoring so in this case correctness means it is indeed it passes the test so for this case if you kind of sanity check it looks right right these are all JSON strings that's good there's no preamble and the value here matches the ground truth so that's good yes yes no yes yes no so that's actually great and we can actually just go back and we can look this was for example open AI so that's cool we can look at an anthropic also that got 100% so that's great now here it's interesting we see chat fireworks llama 13b gets a zero. We actually saw this previously, but we can dig in and see why. Is it because it has this preamble? We don't want that. It's actually the graders doing the right thing. It's catching that and telling us that indeed this should be scored zero as it is. This is actually a really nice trick. You can basically pin an evaluator to a dataset. It can automatically run when you kick off these experiments in the hub, that evaluator just runs behind the scenes for each experiment and automatically scores the results. Really convenient. And of course, I still like to manually review in some cases, but imagine I'm kicking off a large number of experiments. The grader gives me a very quick way to kind of sanity check. So for example if i go back here let's say this was like 20 different experiments on different lms i can use that correctness column from the automated grader to sanity check and say okay which ones look good and bad so i could like drill in and decide which lm i actually want to use for this particular task so the two video the press video and this one kind of pair really well together. It shows how to use LLM-based graders with data sets to automatically perform grading for any experiment you kick off in the hub. All with no code. Very convenient. So I do like to use this when I'm making determinations about what LLM to use, for example, for this grading task, which I've used in a lot of RAG tutorials. I like using LLM graders to sanity check quickly that give me like good candidates to then drill into. So that's really kind of how this can be used together with prompt-based experimentation. Thanks.
Attach evaluators to datasets | LangSmith Evaluations - Part 9
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video shows how to attach evaluators to datasets so that they automatically run when an experiment is performed on the dataset .

Documentation: 
https://docs.smith.langchain.com/evaluation/faq/experiments-app