 Hi, this is Lance from LanxChain. This is the 15th video in our LanxSmith evaluation series. We're going to focus on regression testing. So the past few videos talked a lot about rag evaluation. Just to refresh, for example, we talked about how to evaluate the rag chain answer versus a reference or the answer versus the documents for hallucinations, or even the documents relative to the question for kind of retrieval relevance. Now, here's a common situation that comes up when I think about building a rag pipeline. What if I want to change the LLM? So, you know, a good example is some pretty cool and interesting new open-source LLMs have recently come out, like LLAMA3, PHY3. How can I know whether or not I can replace, for example, GPT-4 as my baseline with one of these for my particular use case in my particular rack pipeline? This is where the notion of regression testing comes in. So we've previously talked about building an eval set, running it on, for example, our answers, and evaluating answer relevance or answer accuracy in this case relative to ground truth answer. Regression testing allows me to run this on a few different examples, like let's say different LLMs that perform the generation step, and to identify cases that either improve or regress relative to my baseline. So the key point of regression testing is the ability to isolate instances in your eval set that are either getting better or worse. So that's kind of the main intuition. So to kick this off, I'm just going to build a new eval set. So that's all defined here. It's going to be called RAG QA LCL. So it's going to use the same line transpression language documentation I used for the prior RAG videos, just a few other, a few slightly different questions. So I've built that and I've built my index. So again, we're just indexing the language expression language documentation. Again, I'm choosing a large chunk size, just like we saw before, no major changes. So I've laid out my RAG pipeline and this looks very similar to what we saw before. The only difference is I've added a slight modification where I can specify different providers. So in this case, I can choose O-Lama, which allows me to run a number of open source LLMs locally, along with OpenAI. And again, just like RAG prompts for the open source models versus OpenAI. But again, no major differences here, really. I'm just extending my RAG model or my RAG chain so I can run with different LLMs. That's all all i'm doing here and i'm defining three of them so i have my rag chain with open ai right here with llama 3 and lever eye chain with 5-3 again llama 3 and 5-3 we use open ai as a provider and i'm just specifying the model i've already downloaded these locally just with simply olama pull llama 3 or a olama pull phi3 and it'll just run locally on my laptop so that's a whole separate thing but that's a pretty cool nice thing about olama so i've defined an evaluator for answers so we saw this previously this labeled score string evaluator basically lets me supply some criteria so in this case you know i'm asking is the assistance answer grounded in the ground truth answer and i give it you know some criteria for scoring one through ten ten being the best one being the worst so that's all all that's going on here and i kick off my eval right so i run eval on open ai first i run eval on llama 3 i run eval on Phi3. So those evals have run. Now let's go over and look at my data set. So here is the data set we created. Again, RABQA LCL, let's just confirm that here. Yeah, so this is the eval set we're running on. And it's the eval set we created up here. So that's right here. So we've created our eval set. Now we can look at our experiments. So this is what we saw before, right? So this is our experiment page. No major changes here, right? We can see our three experiments run here. Now what I'm going to do is I'm going to select these three experiments. So I'm going to go one, two, three, and I'm going to open the comparison mode. This is where things are going to get kind of interesting. We're going to see some things that are a bit new here, and let me just scroll over a little bit so first we're going to see is the three experiments we selected are all here okay and we're seeing this idea of a baseline so what does baseline mean the baseline is basically on reference so in my particular case so let's just say gpd4 was my prior baseline right so gpd4 was a model that I had been using, and I want to see how Phi 3 and Lama 3 compare to GPT-4. So what I can do really easily is, so these are our three examples, just like we saw. So again, I can click on this, and this will basically open up the example. So I can see here's the question, here's the answer. So that's just like, you know, we've kind of gone through that before. So here's the input. Here's the reference output. And now here's my three experiments. And you can see, just like before, we've set a baseline. And I can see the scores from my evaluator here. Now what's interesting is I can scroll over and I can look at my two other experiments. So Lama 3 here, PHY3 here. So what's pretty cool is immediately your attention is drawn to these red or green cells. And this simply means if it's a red cell, it means the performance is worse than my baseline. If it's a green, it means it's better. So in this case, right, my baseline was 0.1. In this case, PHY3 actually for the second example does better. It got a score of 0.5. That's why it's green. Otherwise, you can see there's regressions across the board for questions 1 and 3 for both of the models. Now, I can do a few different things here. First, like we saw before, I can always click on this button, and that actually opens up the evaluator run itself so i can audit the evaluator so that's again we talked about that previously it's really useful you can kind of see why a particular score was given so that's really cool and i can go back and you can see my baseline is still preserved so that's actually really convenient um now this thing at the top tells me the number of total regressions or progressions or improvements relative to my baseline. So I can click this and it'll highlight only the cases that improve. Click this and it'll highlight only the cases that regress. Again likewise here. So again it's really useful and nice way to drill into the cases that are improving or getting worse relative to the baseline. In this case there's only three examples, but in realistic cases, you may have an e-mail set of maybe dozens or hundreds of examples. So this can very easily be used to isolate your progressions or your improvements and your regression. So it's really helpful. I do want to also show here, if you go to the display, you can actually open up additional metrics. This will show you token usage, this will show you token usage. This will show you latencies. So you can kind of close that down to make less or less. You can see the full text of the generations if you want to see that. But again, it's by default removed. So it's like way cleaner. And that's probably the main things I'd like to show you there. You can, of course, choose different experiments or additional experiments if they're not all chosen here. It's easy to add experiments. So anyway, this is a really useful and convenient thing. I do this all the time, so I'm always running experiments with different LLMs. I always want to isolate cases that are improving or regressing, and I want to dig into why. And basically using these filters, I can very easily isolate the cases that are, for example, getting worse, drill into why. Again, in each of these cases, you can see, you actually can look at the trace again. So this will drill into the trace that resulted in the generation. I can go down, I can audit the entire chain. So this is like my chatalama invocation here. This is like the prompt, which contains the context, my rag answer. So again, a lot of things you can play with here, really useful, and regression testing is obviously super powerful for identifying cases that your chain, for example, is getting worse or better with different perturbations, like different models, different chunk sizes, or other things. So definitely play with this. It's extremely useful and helpful in building LLN applications. Thanks.
Regression Testing | LangSmith Evaluations - Part 15
Evaluations can accelerate LLM app development, but it can be challenging to get started. We've kicked off a new video series focused on evaluations in LangSmith.           

With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video focuses on Regression Testing, which lets a user highlight particular examples in an eval set that show improvement or regression across a set of experiments.

Blog: https://blog.langchain.dev/regression-testing/

LangSmith: https://smith.langchain.com/

Documentation: https://docs.smith.langchain.com/evaluation/faq/regression-testing