 Hi this is Lance from Langchain. This is the seventh video in our Langsmith evaluation series. So our first video gave kind of a context as to why evals are interesting and important. The second video talked about Langsmith primitives. Our third video showed how to create manually curated data sets. We built one based upon this Databricks blog post. The fourth one showed how to build data sets for user logs. So if you have an app in production, you want to kind of capture user questions and create a data set from them, you can very easily do that. We talked through that. We then talked about various judges for data sets, so different types of evaluators. We showed how to use a built-in langchain evaluator for question answering. We applied that to our Dataverse data set. And we just talked through custom evaluator so again we've kind of showed this this flow diagram and go to go to those videos if you want to kind of deep dive into those topics now we're going to have a little bit of fun so this is where you know you kind of get into like very real world use cases and where you kind of get into very real world use cases and needs, you often want to do comparisons. So let's ask a really practical question. How does Mistral 7B running locally on my laptop compare to GPT-35 Turbo on this little challenge we've set up? Again, remember, we have a four-question eval set on this Databricks blog post. How does an OpenL undo versus gpd35 turbo so just a little note here i'm using olama for this um just you can download it going to uh alama.com you can do a llama pull mistral to get the model and you can kind of follow instructions here um so here's my setup i'm going to create a new function that does the same thing that we already were doing with open ai but here i'm going using mistral this is running locally on my laptop so again it just i can answer ask and answer questions about the particular blog post so that's answered i just asked a question and here we go so you know the answer is streaming out very good and it's obviously slower than open ai is exactly what we would expect but we really care out here is i want to know about quality how does it actually compare on this little challenge i built for myself so what are we actually doing uh here we would develop a curated data set of four examples on this data bricks blog post um i'm using lm as a judge again remember the built-in cot qa post. I'm using LLM as a judge again. Remember the built-in COTQA evaluator I'm using. And I have ground truth answers for every question. I'm doing an A-B test between GPT-3-5 Turbo and Mistraw running locally on my laptop. So that's the setup. And it's pretty easy. So remember, we've already built or defined this data set dbrx we've already used this devaluator cotqa so that doesn't change at all all that changes is i'm just passing in this new function now now let's go back and look at that function it looks exactly like our old one a few little twists i'm using olam instead of openai that's really it same output object same output object answer you know addict with an answer that's it simple we just saw it work here so what i can do is kick off this eval this will take a little bit because it's running locally i have an m2 mac uh with 32 gigs by the way so that kind of gives you some sense i've heard a lot of people having good results using this draw 7b but on far smaller machines though so it's a really nice open model to work with if you want and you can see it's still churning it's streaming its answers out here it's actually done but didn't take that long it ran against my four questions now here's where it can get really interesting let's go over to my data set now i can see here that there's three runs so this is our initial run uh or experiment you can think of, with OpenAI. This is that second one to do with a custom evaluator. We're not interested in that. That was just kind of a quick more unit test that wasn't a proper kind of a LLM-based evaluation. And now here's our latest one. So here's where it gets really interesting. I can click on this and this. So my mistral run my open i run and i can click this compare thing that opens up this compare mode you can already see some nice statistics here so what i can see is like average score so the first run which was open ai indeed does quite a bit better in terms of score latency as exactly we would expect mistraw is slower by quite a bit um and here's the latency distribution and so forth so you get some immediate statistics about the two runs now here's where i've done a lot of work in the past and you know this is kind of the crux of ab testing that's extremely useful uh i mean that's why it's very helpful to do this inside lengths but this is all kind of the crux of A-B testing that's extremely useful. I mean, that's why it's very helpful to do this inside Lanks, but if this is all kind of captured for you, managing this all yourself can be quite painful. Here's my first question. Here's the reference output. Here is the output returned by Mistrawl. Here's the output returned by OpenAI. So I can actually look at those in detail. I can kind of zoom in, look at the answers, and like, hey, they look very similar here that's really cool and you can see my grader also assesses them both as correct and again we talked about you can actually click on the evaluator run for each case to see why but they look good now here's where it gets a little interesting it looks like my mistraw running locally is is did not receive a correct grade um and opening i did so let's actually look at what was the question what is the context window of the drbx a drop model okay so it's 32k token right what did mistraw think uh context window is 2048 so that is definitely wrong and we would have to investigate as to why Mistral believed that to be the case. It could be, you know, there could be many different reasons why I failed for that one. But indeed, our grader is correctly grading that as a wrong response. For fun, we can actually dig in and look at that particular grading trace. And we can see why Susan's answer is incorrect the student states that the context window is 2048 um the context says clearly 32k there you go so the grader is doing the same thing and we can kind of go through each one so this is like a toy example but it shows you a very important useful concept of comparative a b testing so like you might want to compare different prompts different lms and this is a very nice way to do that you can see it's extremely simple we've just supplied our data set name um so we're of course running against the same data set where you know typically i like to apply different experimental prefix to to enumerate the different experiments i'm running so that's easy you can also capture that in metadata by the way so that's another way to kind of differentiate your experiments. And I'm using the same grader, of course. And I'm just modifying my function, which in this case was just, hey, swapping out Mistral, swapping in Mistral for OpenAI. So again, this just shows you how to use this compare mode in Lanksmith to do A-B testing really nicely. In this particular case, we're comparing Mistral versus OpenAI. We can look at, you know, kind of overall run statistics as well as granular answer-wise differences. We can inspect the grader as shown here. We can look at the runs as shown here. So this gives you a very flexible general place to do a b testing uh across different parameters in this case i use different uh different llms um and i've used this all the time for lots of different kind of projects and it is indeed quite useful it's very nice to have this all kind of managed in one place for you um so we're going to be kind of diving into some deeper themes after this this is kind of the final video of our like kind of introductory concepts so we kind of zoom all the way back out what do we talk through we just talked through manual building your manually curated data set in this case running lm as a judge against the ground truth for a b testing so we kind of went through that we also had talked through um you know same setup but basically just simple unit test using custom evaluators we talked through um yep we talked through yeah different data set creation yeah we talked through here we go ln as a judge with ground truth um so for like you know uh just just uh you know ln is greater evaluation um but no ab tests so in that case it's just looking at like a simple model and evaluating it using LM as the judge. We talked about the information flow for evaluation. We talked about different ways to build data sets from user logs, from manual curation. And that's really it. This gives you kind of like the groundwork you need to do a lot of different things. Build a custom evaluators and A-B testing is frankly covers a huge service area of use cases. And we're going to do some deep dives following this. So stay tuned for additional videos. Thanks.
Eval Comparisons | LangSmith Evaluations - Part 7
With the rapid pace of AI, developers are often faced with a paradox of choice: how to choose the right prompt, how to trade-off LLM quality vs cost? Evaluations can accelerate development with  structured process for making these decisions. But, we've heard that it is challenging to get started. So, we are launching a series of short videos focused on explaining how to perform evaluations using LangSmith.

This introduces how to use the LangSmith UI to compare (e.g., different prompts, LLMs, etc) across a dataset. 

Documentation:
https://docs.smith.langchain.com/user_guide#comparison-view

Code:
https://github.com/langchain-ai/langsmith-cookbook/blob/main/introduction/langsmith_introduction.ipynb