 Hi, this is Lance from LangChain. We're continuing our LangSmith evaluation series focused on backtesting today. So to motivate this, let's say we had an app in production. Say, for example, it's one of our RAG apps that we kind of talked about in some prior videos. And the particular RAG app in our case is GPD4 Turbo using a vector store. So that's in production. We're collecting production traffic from users using this app version. Now, what happens if we wanna try different variants? Like let's say I wanna try context stuffing rather than using my Vector Store. One really convenient and nice thing to be able to do there is take a bunch of production logs we've already collected and just run my new app variant on them and see how the output compares, right? That's a really common thing you might want to do. So if you look at our framework here, we're talking about the data set. Actually, you can come from existing production logs. That's really convenient. We don't have to build some, like, curated data set from it. We can just take user data that actually has been, you know, contributed to our existing app and turn that into a data set. So that's step one. And we can test, then, a different app variant on those same inputs. So that's step one. And we can test then a different app variant on those same inputs. So that's really useful. And we can use like a pairwise evaluator to compare our new variant of the app to our old or production version that's running currently. So that's like a common workflow. Now let's walk through this. So I'm going to create a new project. I'll call it backtesting. I'll create a new one here, backtesting, let's say V2. And here's my app. So let's say this is running in production. Here's a few user questions related to language and expression language. So if you call this particular app is actually ingesting information about language and expression language. And so these are all running. Cool. So we can just kick a few of these off Great. So I've run five different user questions through my app. Now, assume this is, you know, obviously we're doing it in the notebook, but assume this is a production app. It's just out there. These are users interacting with your app, right? So that's what we're simulating here. Now, I've created this project back to S&2, we can go over to Langsmith, I can look at my projects page, and it's here. And so great. So here's all my traces, they're all logged here. So that's fantastic. Now what I can do is this code right here, we'll take these traces. So I can do is this code right here will take these traces so I can specify my run filters in terms of start and end time. And I can choose the project that I want to grab from. So in this case, it's this project we just logged to. And basically what we can do here is I can create a new data set. So I'm taking these user logs effectively right here. And what I'm doing is I'm turning them into a new data set. So now we'll see if you go over to our data sets page we have this new data set called backdressing v2 and it has like the exact time and so forth and what you'll see here is pretty nice those user inputs are now just simply inputs in my data set so I've kind of sucked them into a data set that's all I've done now what I can do here is I can run an evaluation on this data that we just created using a different variant to my app. So I'll call this predict rag answer GPT-4 turbo. It's a different app variant. This particular variant doesn't use a vector store. It does context stuffing, which we can talk about a little bit later. But let's say I kick off that evaluation. So I run that right now. So if I can go back to my dataset here, we can see a new experiment has been kicked off. And what's pretty cool about this, when I create this dataset, my initial experiment is the prod baseline. So that's basically my production app that we've been using that we logged from. And that is actually what we collected our inputs and our outputs initially from. So this is kind of like our baseline, right? This collected our inputs and our outputs initially from. So this is kind of like our baseline, right? This is our inputs. These are outputs from that baseline version we looked at initially. Now, I've just run a new experiment using our variant, right? So this is a GPT-4 Turbo. So that's what I just kicked off. And this is running now. You can see it's still running. You can see kind of the name here, my experiment prefix GPT-4 Turbo. And that is what you can see kind of the name here, my experiment prefix GPT-4 Turbo, and that is what you can see here. So GPT-4 Turbo, that's great. So that's all running, and we can check and see the state of that experiment. It looks like it's still going. Cool. So there's one more to go, And it looks like it's done. So that's great. So now we are finished. So here's where things get kind of interesting. So I've run. So let me just back up. What do we do here? So if I go all the way back, first I had a project that I was logging to. So this is simulating, for example, like a production app. And these are user interactions that are being logged to some particular project. What I've done is I've taken these input and output pairs, and I've pulled them into a new data set. And you can see that right here. I suck those in. And by default, when I do that, that baseline is grabbed as my prod baseline. So this is what my production app inputs and outputs. Now what's cool is I was able to just run a new experiment with my new app version on those same user inputs, but now I have my new output. So these are the outputs from my new chain. That's great. Now this all comes together in kind of a cool way because I can use a pairwise evaluator to compare them. So I just created a new pairwise evaluation prompt for this particular task. And I think I already have it open here, great. So this particular task, if you recall, is code question answering based on the language and expression language documentation. So I set up my pairwise prompt and I just say, previous acts as a judge to evaluate the quality of code responses from two assistants related to language, I give my criteria here, begin your evaluation comparing the responses based upon, do they contain a clear overview of the problem? Do they contain code imports? Do they contain code solution, right? So you don't allow length to affect the results and so forth. So anyway, that's my evaluation prompt. Now I've defined, I can define this pairwise evaluator. We just already talked about this in the last video. So you've already seen a good example of that. I'm creating a pairwise evaluator right here. And what's pretty cool is I can run that pairwise evaluator on my two experiments. So this is my, this is the name of my most recent experiment. It's GPT four turbo boom. And my initial experiment, I can go back and get that name. So it's gpt4 turbo boom and my initial experiment i can go back and get that name so it's this prod baseline so i can just plug those in okay so i'm running a comparative evaluation between my prod baseline and my new app variant that i ran back testing on i kicked that, so now this is running. And of course, it's gonna be logged to the same overall dataset here. So if I go back to my dataset, recall we had two experiments, my prod baseline, my backtesting, which is running the same prod inputs with my new chain, GPT-4 Turbo with context stuffing. And now I've kicked off an experiment that's comparing the two. So it's comparing my prod baseline versus my new variant, GPT-4 Turbo with context stuffing, versus the baseline, if you recall, is actually using retrieval from a vector store. The variant is actually using GPT-4 Turbo with context stuffing the entire Lcell docs into the into the lm which has some benefits over retrieval because you don't have issues related to retrieval of the correct document chunks you're just passing in everything and so there's some benefits there that we could talk about at length a different time so this is running and it looks like it might be finished this is done and this is pretty cool. So I can go here, I can look at here's all the inputs, here is my prod baseline, here is my variant. This is what I ran backtesting on. And what's pretty neat about the pairwise value, we can really see the preference. And so in this particular case, it prefers the output from our variant over the baseline. And again, just as we saw in the previous video we can actually click into here and investigate why so you can go to the evaluator and it actually gives you an explanation as to why so anyway you can really dig into this this is a pretty nice workflow so if we zoom all the way back out what do we really do here uh well we had an example of an app running a production. This is kind of a simulation of that. We collected five user inputs. We turned those into a data set. That's what we did here. We then ran what we call backtesting on that data set with a new app variant. So like I want to test a new app version. In this case, my baseline used retrieval from a vector store. My new app variant used context stuffing. So I ran that evaluation. And so then that resulted in a generation. So that's right here. For every input, I produced a new generation or output from my variant. That's great. And then I ran this comparative eval saying, hey, which one's better? Here's my prompt. And I kicked that off. I just added the names of the two experiments. And as we saw before, you get this pretty nice comparative assessment with detailed evaluation. So pretty useful thing to do. If you have any app running in production, you can pretty easily grab those production logs, turn them into a data set, do back testing on them with like different variants, different chains you want to test. And you can even do those production logs, turn them into a data set, do back testing on them with like different variants, different chains you want to test. And you can even do things like pairwise evaluation to say, hey, which one's better or worse based on some criteria that I can define. So really nice workflow, highly convenient for kind of testing at, testing different variants of chains that you want to, do you want to put in production? Thanks.
Backtesting  | LangSmith Evaluations - Part 19
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This video focuses on Backtesting. We show you how to build a dataset directly from production logs. We then show how to run different versions of your app on those logs and use pairwise evaluation to compare them against the baseline production app.