 Hi, this is Lance from LangChain. This is the 13th part in our LangSmith evaluation series. We've been talking about rag evaluation. In the last video, we saw how to do comparison of my LLM-generated answer to a reference answer. We just kind of dove into that. Now let's actually talk about some of the other types of evaluation we can do. In particular, let's talk about hallucination evaluation. So this is basically an evaluation of our retrieved documents relative to our answer. So you recall from before, we build a rag chain, we have a retriever here, here's our chain, and the chain returns both answer and contexts. Okay, so that's kind of the setup. We just talked through, basically again, reference answer evaluations. This is what we went through. Now let's talk about what I'm going to call type two. So answer hallucination. through. Now let's talk about what I'm going to call type two. So answer hallucination. So here, actually, we can reuse a lot of the same thinking and a lot of the same components. Again, before we use a langchain string evaluator, because we're doing string comparisons fundamentally, and I previously showed that we can use the CUT, the chain of thought QA evaluator for answer evaluations. Now, in this case, we're going to change that up a little bit because we want to do comparison now between our answer and the reference. So this is basically the reference documents. Okay. So this is kind of an internal comparison for if something is present in the answer that's not in the documents, we want to know, we want to penalize that. A common thing that happens here is hallucinations, for example. So all we're doing is our rag chain returns both answer and context, and we're going to pass those into our evaluator. Again, our answer will be the prediction, and we're going to use our context now as the reference. So we're basically doing comparison between our answer and the reference being the retrieved documents. That's really all that's happening. Now here, instead of using the COTQA evaluator, I'm going to use the criteria evaluator. So it's another option that you can use, and you'll see the criteria is kind of nice because it allows us to supply some kind of custom criteria to the grading process. And we just talked through the information flow. Now here's really the crux of it. You'll look, this is actually very familiar to what we saw before. It's just another line chain string evaluator, just like before. In this case, it's a slightly different name. So it's called labeled score string. And the key point here is I can pass a particular criteria that I want to evaluate those strings on to the evaluator. Now, it's still LLM as judge, just like before. But in this case, I have this custom criteria field that I can pass in. So here's where I can kind of add all the kind of rules or kind of logic that I want the evaluator to follow. So here I'm going to say, is the assistant's answer grounded in the ground truth documentation, right? And I tell it like what a score of zero means. I tell it what a score of five means. A score of five is the answer contains some information that's not in the document. So some hallucinations. Zero is like it's all hallucination. Ten is that it's perfect, okay? Now this normalize thing just lets me like normalize the scores between zero and one as a convenience. So it's basically gonna take the score returned by the greater will be zero to 10. I normalize that by 10 to produce zero to one. Now here is where we saw before. This is where I hook up my run and my data set outputs to the inputs of my prompt. So this is the key part. So we can look at this here. So in this particular case, my prediction is going to be the answer, just like before. is going to be the answer, just like before. The reference is actually just going to be my run context or my retrieved documents. And the input is just the example input. So that's actually not very important for this eval, but we'll just keep it in there. But the key point is this. My prediction is my answer, and my reference are the retrieved documents. So that's all that's happening. So I can define that. Now I kick that off. I'll add an experiment prefix here to note that it's hallucination grading. And that's really it. So this is kicked off. I can go ahead to my – so I've just kicked off evaluation. And now that's run. And I can go over to my data set I've just kicked off evaluation and now that's run. And I can go over to my data set and I can look at the results. So here's again, that hallucination, yeah, prefix that we added and I can open this up. So now I can see the scoring of all my runs here. And again, this is looking at the answer relative to the retrieved documents. So it's kind of a hallucination score. And so the results are kind of mixed. It looks like in one case, it gives it a score of 0.5. One case, it gives it a score of perfect. I can look at, for example, the bad case here. I can open up that run. So here we go. We can go down and we can actually look at, this is of course the prompt. So it has all the retrieved documents and it also has the answer. And then it has the LLM reasoning. So the assistant provide detailed comprehensive response, but it found something it didn't like, it gives it a two. So anyway, the key point is this shows you how to hook everything up. You can see all we did was we basically hooked up our chain answer and context into our evaluator reference and prediction, and then we just let evaluator run. We give it instructions here in this criteria field, and it's all logged to Langsmith. We can see the scores here. So anyway, it's a pretty nice trick, and definitely encourage you to look into using this criteria evaluation with labeled score string.
RAG Evaluation (Answer Hallucinations) | LangSmith Evaluations - Part 13
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video focuses on RAG (Retrieval Augmented Generation). We show you how to check that your outputs are grounded in the retrieved documents of your RAG pipeline. You can use LangSmith to create a set of test cases, run an evaluation against retrieved documents, and dive into output traces – helping you ensure your responses are hallucination-free.

Documentation:
https://docs.smith.langchain.com/cookbook/testing-examples/rag_eval#type-2-answer-hallucination